oVirt支持虚拟机的HA配置, 当集群中有合适的机器可以接管failed的宿主机上的虚拟机时, 这些虚拟机会自动在同集群下的合适的主机上运行. 但是有几个前提条件 :
还可以使用主机的电源管理, restart来看看是否可以重启服务器.
配置虚拟机, 打开虚拟机标签, 打开高级选项, 配置Host. 允许在集群中的任何主机运行.
[参考]
1. Power management must be configured for the hosts running the highly available virtual machines.
需要配置HA的虚拟机, 它的宿主机的电源管理必须配置.
2. The host running the highly available virtual machine must be part of a cluster which has other available hosts.
需要HA的虚拟机, 所在的cluster必须有其他主机可以运行这个虚拟机. 并且主机必须是运行状态.
3. The destination host must be running.
4. The source and destination host must have access to the data domain on which the virtual machine resides.
目标主机和当前运行虚拟机的主机, 必须同时可以访问数据域(如存放虚拟机的DISK IMG domain)
5. The source and destination host must have access to the same virtual networks and VLANs.
网络方面, 目标主机和当前的宿主机必须有同样的虚拟网络(网桥)和VLAN.
6. There must be enough CPUs on the destination host that are not in use to support the virtual machine's requirements.
资源方面, 目标主机必须有足够的CPU资源运行failover的虚拟机.
7. There must be enough RAM on the destination host that is not in use to support the virtual machine's requirements.
内存资源也是如此.
配置比较简单 :
首先查看集群中的主机, 切换只会发生在同一个集群的主机之间.
主机配置, 首先要配置管理卡, 包括用户角色, 开启LAN ipmi, 允许开关机等. (一般服务器都有WEB界面配置)
配置电源管理, 使用Test 测试电源管理是否正常.
同时迁移选项要允许手工和自动迁移.
配置HA, 勾选Highly Available.
测试虚拟机failover, 连接到当前运行虚拟机的主机, 把网卡停掉.
ifdown em1, 观察日志
/var/log/ovirt-engine/engine.log, 看看会不会自动将虚拟机切换到另一台主机.
切换前应该会fence当前的宿主机. (通过配置的电源管理fence设备进行的)
运行3台虚拟机大概3分钟切换完. 这个应该是可以配置的, 当然也不能配置太短, 防止因抖动产生的切换.
有问题的话, 查看日志
/var/log/ovirt-engine/engine.log
2014-08-08 17:40:49,203 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-10) domain 8f6f0ec3-2183-48cd-9ec4-aface2540629:gfs1 in problem. vds: 150
2014-08-08 17:41:04,464 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-40) Domain 8f6f0ec3-2183-48cd-9ec4-aface2540629:gfs1 recovered from problem. vds: 150
2014-08-08 17:41:04,465 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-40) Domain 8f6f0ec3-2183-48cd-9ec4-aface2540629:gfs1 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer.
2014-08-08 17:43:14,386 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-90) Command ListVDSCommand(HostName = 40, HostId = b886705b-1e3f-4fc9-82c7-eaf4e5667adc, vds=Host[40,b886705b-1e3f-4fc9-82c7-eaf4e5667adc]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException
2014-08-08 17:43:14,391 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-90) Failed to refresh VDS , vds = b886705b-1e3f-4fc9-82c7-eaf4e5667adc : 40, VDS Network Error, continuing.
java.util.concurrent.TimeoutException
2014-08-08 17:43:19,197 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command GetCapabilitiesVDSCommand(HostName = 40, HostId = b886705b-1e3f-4fc9-82c7-eaf4e5667adc, vds=Host[40,b886705b-1e3f-4fc9-82c7-eaf4e5667adc]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-08-08 17:43:19,203 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-65) Failed to refresh VDS , vds = b886705b-1e3f-4fc9-82c7-eaf4e5667adc : 40, VDS Network Error, continuing.
java.net.NoRouteToHostException: No route to host
2014-08-08 17:43:24,234 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-84) Command GetCapabilitiesVDSCommand(HostName = 40, HostId = b886705b-1e3f-4fc9-82c7-eaf4e5667adc, vds=Host[40,b886705b-1e3f-4fc9-82c7-eaf4e5667adc]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
2014-08-08 17:43:24,237 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-84) Server failed to respond, vds_id = b886705b-1e3f-4fc9-82c7-eaf4e5667adc, vds_name = 40, vm_count = 3, spm_status = None, non-responsive_timeout (seconds) = 61, error = java.net.SocketTimeoutException: connect timed out
2014-08-08 17:43:24,267 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out
at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:102)
at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56)
at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31)
at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:511)
at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:488)
at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:344)
at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:231)
at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618)
at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288)
at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227)
at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151)
at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143)
at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56)
at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167)
at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137)
at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126)
at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140)
at com.sun.proxy.$Proxy54.getVdsCapabilities(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:225)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
, Custom Event ID: -1, Message: Host 40 is non responsive.
2014-08-08 17:43:24,294 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-6-thread-36) ResourceManager::vdsNotResponding entered for Host b886705b-1e3f-4fc9-82c7-eaf4e5667adc, 172.16.3.40
2014-08-08 17:43:24,337 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-6-thread-36) [4e0cec7f] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: b886705b-1e3f-4fc9-82c7-eaf4e5667adc Type: VDS
2014-08-08 17:43:24,375 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-6-thread-36) [4e0cec7f] Opening SSH Soft Fencing session on host 172.16.3.40
2014-08-08 17:43:25,199 ERROR [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-6-thread-36) [4e0cec7f] SSH Soft Fencing command failed on host 172.16.3.40: Failed to get the session.
Stdout:
Stderr:
Stacktrace: org.apache.sshd.common.RuntimeSshException: Failed to get the session.: org.apache.sshd.common.RuntimeSshException: Failed to get the session.
at org.apache.sshd.client.future.DefaultConnectFuture.getSession(DefaultConnectFuture.java:43)
at org.ovirt.engine.core.utils.ssh.SSHClient.connect(SSHClient.java:294) [utils.jar:]
at org.ovirt.engine.core.bll.utils.EngineSSHClient.connect(EngineSSHClient.java:59) [bll.jar:]
at org.ovirt.engine.core.bll.SshSoftFencingCommand.executeSshSoftFencingCommand(SshSoftFencingCommand.java:77) [bll.jar:]
at org.ovirt.engine.core.bll.SshSoftFencingCommand.executeCommand(SshSoftFencingCommand.java:44) [bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1133) [bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1218) [bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1894) [bll.jar:]
at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:]
at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:]
at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1238) [bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:351) [bll.jar:]
at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:415) [bll.jar:]
at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:394) [bll.jar:]
at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:636) [bll.jar:]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65]
at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65]
at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:211) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:363) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:194) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:173) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view10.runInternalAction(Unknown Source) [bll.jar:]
at org.ovirt.engine.core.bll.VdsEventListener$2.run(VdsEventListener.java:154) [bll.jar:]
at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:97) [utils.jar:]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_65]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_65]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_65]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_65]
Caused by: java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65]
at org.apache.mina.transport.socket.nio.NioSocketConnector.finishConnect(NioSocketConnector.java:223) [mina-core.jar:]
at org.apache.mina.transport.socket.nio.NioSocketConnector.finishConnect(NioSocketConnector.java:45) [mina-core.jar:]
at org.apache.mina.core.polling.AbstractPollingIoConnector.processConnections(AbstractPollingIoConnector.java:427) [mina-core.jar:]
at org.apache.mina.core.polling.AbstractPollingIoConnector.access$500(AbstractPollingIoConnector.java:62) [mina-core.jar:]
at org.apache.mina.core.polling.AbstractPollingIoConnector$Connector.run(AbstractPollingIoConnector.java:476) [mina-core.jar:]
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) [mina-core.jar:]
... 3 more
2014-08-08 17:43:25,292 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-6-thread-36) [63bd815a] Lock Acquired to object EngineLock [exclusiveLocks= key: b886705b-1e3f-4fc9-82c7-eaf4e5667adc value: VDS_FENCE
, sharedLocks= ]
2014-08-08 17:43:25,415 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host 150 from data center Default was chosen as a proxy to execute Restart command on Host 40.
2014-08-08 17:43:25,417 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-6-thread-36) Using Host 150 from data center Default as proxy to execute Restart command on Host 40
2014-08-08 17:43:25,423 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-6-thread-36) Running command: VdsNotRespondingTreatmentCommand internal: true. Entities affected : ID: b886705b-1e3f-4fc9-82c7-eaf4e5667adc Type: VDS
2014-08-08 17:43:25,487 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-6-thread-36) [b647a7a] Running command: StopVdsCommand internal: true. Entities affected : ID: b886705b-1e3f-4fc9-82c7-eaf4e5667adc Type: VDS
2014-08-08 17:43:25,514 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-36) [b647a7a] START, SetVdsStatusVDSCommand(HostName = 40, HostId = b886705b-1e3f-4fc9-82c7-eaf4e5667adc, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 15777d88
2014-08-08 17:43:25,518 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-36) [b647a7a] FINISH, SetVdsStatusVDSCommand, log id: 15777d88
2014-08-08 17:43:25,637 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [b647a7a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host 150 from data center Default was chosen as a proxy to execute Stop command on Host 40.
2014-08-08 17:43:25,638 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-6-thread-36) [b647a7a] Using Host 150 from data center Default as proxy to execute Stop command on Host 40
2014-08-08 17:43:25,663 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-6-thread-36) [b647a7a] Executing <Stop> Power Management command, Proxy Host:150, Agent:ipmilan, Target Host:40, Management IP:172.16.3.192, User:sky, Options:
2014-08-08 17:43:25,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-36) [b647a7a] START, FenceVdsVDSCommand(HostName = 150, HostId = 44379bb8-e87b-4e00-a18e-5df43922da82, targetVdsId = b886705b-1e3f-4fc9-82c7-eaf4e5667adc, action = Stop, ip = 172.16.3.192, port = , type = ipmilan, user = sky, password = ******, options = ''), log id: 6ca5cf12
2014-08-08 17:43:25,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-36) [b647a7a] FINISH, FenceVdsVDSCommand, return: Done, log id: 6ca5cf12
2014-08-08 17:43:25,856 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-6-thread-36) [b647a7a] Waiting for vds 40 to stop
2014-08-08 17:43:30,858 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-6-thread-36) [b647a7a] Attempt 1 to get vds 40 status
2014-08-08 17:43:30,955 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [b647a7a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host 150 from data center Default was chosen as a proxy to execute Status command on Host 40.
2014-08-08 17:43:30,956 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-6-thread-36) [b647a7a] Using Host 150 from data center Default as proxy to execute Status command on Host 40
2014-08-08 17:43:30,981 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-6-thread-36) [b647a7a] Executing <Status> Power Management command, Proxy Host:150, Agent:ipmilan, Target Host:40, Management IP:172.16.3.192, User:sky, Options:
2014-08-08 17:43:31,005 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-36) [b647a7a] START, FenceVdsVDSCommand(HostName = 150, HostId = 44379bb8-e87b-4e00-a18e-5df43922da82, targetVdsId = b886705b-1e3f-4fc9-82c7-eaf4e5667adc, action = Status, ip = 172.16.3.192, port = , type = ipmilan, user = sky, password = ******, options = ''), log id: 79e33fc1
2014-08-08 17:43:31,177 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-36) [b647a7a] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 79e33fc1
2014-08-08 17:43:31,178 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-6-thread-36) [b647a7a] vds 40 status is off
2014-08-08 17:43:31,180 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-6-thread-36) [b647a7a] START, SetVmStatusVDSCommand( vmId = 77195f8a-787f-4d55-a9d5-932bd5b29d4a, status = Down, exit status = Error), log id: 71d6837d
2014-08-08 17:43:31,185 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-6-thread-36) [b647a7a] FINISH, SetVmStatusVDSCommand, log id: 71d6837d
2014-08-08 17:43:31,263 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [b647a7a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Vm v1 was shut down due to 40 host reboot or manual fence
2014-08-08 17:43:31,268 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-36) [5c9e5d8b] Running command: ProcessDownVmCommand internal: true.
2014-08-08 17:43:31,286 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-6-thread-36) [5c9e5d8b] START, SetVmStatusVDSCommand( vmId = 3248d236-eab3-4e3b-9fb1-325c6dd0fb67, status = Down, exit status = Error), log id: 6d4cd087
2014-08-08 17:43:31,290 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-6-thread-36) [5c9e5d8b] FINISH, SetVmStatusVDSCommand, log id: 6d4cd087
2014-08-08 17:43:31,322 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [5c9e5d8b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Vm c6.5_1 was shut down due to 40 host reboot or manual fence
2014-08-08 17:43:31,324 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-36) [39e060fe] Running command: ProcessDownVmCommand internal: true.
2014-08-08 17:43:31,334 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-6-thread-36) [39e060fe] START, SetVmStatusVDSCommand( vmId = 4263a88b-3a5b-4210-94f8-4b9d958c041b, status = Down, exit status = Error), log id: 7f6e1465
2014-08-08 17:43:31,338 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-6-thread-36) [39e060fe] FINISH, SetVmStatusVDSCommand, log id: 7f6e1465
2014-08-08 17:43:31,367 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [39e060fe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Vm test1 was shut down due to 40 host reboot or manual fence
2014-08-08 17:43:31,369 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-36) [793b3360] Running command: ProcessDownVmCommand internal: true.
2014-08-08 17:43:31,403 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (org.ovirt.thread.pool-6-thread-36) [793b3360] START, UpdateVdsVMsClearedVDSCommand(HostName = 40, HostId = b886705b-1e3f-4fc9-82c7-eaf4e5667adc), log id: b0b116d
2014-08-08 17:43:31,405 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (org.ovirt.thread.pool-6-thread-36) [793b3360] FINISH, UpdateVdsVMsClearedVDSCommand, log id: b0b116d
2014-08-08 17:43:31,413 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [793b3360] Correlation ID: b647a7a, Job ID: 9d592565-238e-46b1-aa69-be53616fb985, Call Stack: null, Custom Event ID: -1, Message: Host 40 was stopped by Engine.
2014-08-08 17:43:31,482 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-6-thread-36) [1e64c05e] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: b886705b-1e3f-4fc9-82c7-eaf4e5667adc Type: VDS
2014-08-08 17:43:31,639 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host 150 from data center Default was chosen as a proxy to execute Status command on Host 40.
2014-08-08 17:43:31,640 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-6-thread-36) Using Host 150 from data center Default as proxy to execute Status command on Host 40
2014-08-08 17:43:31,644 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-6-thread-36) Running command: StartVdsCommand internal: true. Entities affected : ID: b886705b-1e3f-4fc9-82c7-eaf4e5667adc Type: VDS
2014-08-08 17:43:31,669 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-36) START, SetVdsStatusVDSCommand(HostName = 40, HostId = b886705b-1e3f-4fc9-82c7-eaf4e5667adc, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1bc082c7
2014-08-08 17:43:31,672 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-36) FINISH, SetVdsStatusVDSCommand, log id: 1bc082c7
2014-08-08 17:43:31,806 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host 150 from data center Default was chosen as a proxy to execute Start command on Host 40.
2014-08-08 17:43:31,807 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-6-thread-36) Using Host 150 from data center Default as proxy to execute Start command on Host 40
2014-08-08 17:43:31,831 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-6-thread-36) Executing <Start> Power Management command, Proxy Host:150, Agent:ipmilan, Target Host:40, Management IP:172.16.3.192, User:sky, Options:
2014-08-08 17:43:31,855 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-36) START, FenceVdsVDSCommand(HostName = 150, HostId = 44379bb8-e87b-4e00-a18e-5df43922da82, targetVdsId = b886705b-1e3f-4fc9-82c7-eaf4e5667adc, action = Start, ip = 172.16.3.192, port = , type = ipmilan, user = sky, password = ******, options = ''), log id: 381abf8
2014-08-08 17:43:31,994 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-36) FINISH, FenceVdsVDSCommand, return: Done, log id: 381abf8
2014-08-08 17:43:32,002 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (DefaultQuartzScheduler_Worker-4) [5b72f572] START, IsVmDuringInitiatingVDSCommand( vmId = 77195f8a-787f-4d55-a9d5-932bd5b29d4a), log id: 38b509bc
2014-08-08 17:43:32,004 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (DefaultQuartzScheduler_Worker-4) [5b72f572] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 38b509bc
2014-08-08 17:43:32,021 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-6-thread-36) Waiting for vds 40 to start
2014-08-08 17:43:32,041 INFO [org.ovirt.engine.core.bll.RunVmCommand] (DefaultQuartzScheduler_Worker-4) [5b72f572] Running command: RunVmCommand internal: true. Entities affected : ID: 77195f8a-787f-4d55-a9d5-932bd5b29d4a Type: VM
2014-08-08 17:43:32,172 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (DefaultQuartzScheduler_Worker-4) [5b72f572] START, CreateVmVDSCommand(HostName = 150, HostId = 44379bb8-e87b-4e00-a18e-5df43922da82, vmId=77195f8a-787f-4d55-a9d5-932bd5b29d4a, vm=VM [v1]), log id: 703d8622
2014-08-08 17:43:32,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (DefaultQuartzScheduler_Worker-4) [5b72f572] START, CreateVDSCommand(HostName = 150, HostId = 44379bb8-e87b-4e00-a18e-5df43922da82, vmId=77195f8a-787f-4d55-a9d5-932bd5b29d4a, vm=VM [v1]), log id: 4afbd1c
2014-08-08 17:43:32,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (DefaultQuartzScheduler_Worker-4) [5b72f572] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=1024,kvmEnable=true,smp=4,vmType=kvm,emulatedMachine=rhel6.5.0,keyboardLayout=en-us,memGuaranteedSize=1024,pitReinjection=false,nice=0,display=qxl,smartcardEnable=false,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,maxVCpus=160,timeOffset=0,transparentHugePages=true,vmId=77195f8a-787f-4d55-a9d5-932bd5b29d4a,devices=[{address={bus=0x00, domain=0x0000, slot=0x02, type=pci, function=0x0}, specParams={ram=65536, vram=32768, heads=1}, device=qxl, type=video, deviceId=41f9e12f-ef20-483e-a8f2-2c15420e3b14}, {shared=false, iface=ide, index=2, address={unit=0, bus=1, target=0, controller=0, type=drive}, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=49a63040-1d3c-4954-9755-c669324d7c82}, {shared=false, index=0, volumeID=625ca5c4-4252-4b59-9d85-3ca15c29c17a, propagateErrors=off, format=raw, type=disk, iface=virtio, bootOrder=1, address={bus=0x00, domain=0x0000, slot=0x06, type=pci, function=0x0}, domainID=1cf9c0d6-d04c-4d65-a3bb-3e0ed25ee948, imageID=4b47aac8-d8a1-4b59-b723-d259937919ce, specParams={}, optional=false, device=disk, poolID=00000002-0002-0002-0002-0000000000ec, readonly=false, deviceId=4b47aac8-d8a1-4b59-b723-d259937919ce}, {nicModel=pv, address={bus=0x00, domain=0x0000, slot=0x03, type=pci, function=0x0}, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:9a:38:1e, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=b0c90664-7558-4c24-a434-1f300758e3f5}, {specParams={model=virtio}, device=memballoon, type=balloon, deviceId=70ab1f29-159a-4d81-afdd-4adee78c6c60}, {index=0, model=virtio-scsi, address={bus=0x00, domain=0x0000, slot=0x04, type=pci, function=0x0}, specParams={}, device=scsi, type=controller, deviceId=377ccc6c-b6f2-46cb-b5f9-ab4bd2f8d0f7}],acpiEnable=true,vmName=v1,cpuType=Nehalem,custom={device_c8c2ea54-7116-47e0-ab6d-a6019b8b6e61=VmDevice {vmId=77195f8a-787f-4d55-a9d5-932bd5b29d4a, deviceId=c8c2ea54-7116-47e0-ab6d-a6019b8b6e61, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=1, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel0, customProperties={}, snapshotId=null}, device_c8c2ea54-7116-47e0-ab6d-a6019b8b6e61device_c361fa4f-328a-40ff-95bf-bb5511367ed2device_fbd2ef8c-895a-44de-be3c-2d12c9735707device_3989012d-70d6-4125-a6a6-b456a8d6c03b=VmDevice {vmId=77195f8a-787f-4d55-a9d5-932bd5b29d4a, deviceId=3989012d-70d6-4125-a6a6-b456a8d6c03b, device=ide, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x01, function=0x1}, managed=false, plugged=true, readOnly=false, deviceAlias=ide0, customProperties={}, snapshotId=null}, device_c8c2ea54-7116-47e0-ab6d-a6019b8b6e61device_c361fa4f-328a-40ff-95bf-bb5511367ed2device_fbd2ef8c-895a-44de-be3c-2d12c9735707=VmDevice {vmId=77195f8a-787f-4d55-a9d5-932bd5b29d4a, deviceId=fbd2ef8c-895a-44de-be3c-2d12c9735707, device=spicevmc, type=CHANNEL, bootOrder=0, specParams={}, address={port=3, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel2, customProperties={}, snapshotId=null}, device_c8c2ea54-7116-47e0-ab6d-a6019b8b6e61device_c361fa4f-328a-40ff-95bf-bb5511367ed2=VmDevice {vmId=77195f8a-787f-4d55-a9d5-932bd5b29d4a, deviceId=c361fa4f-328a-40ff-95bf-bb5511367ed2, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=2, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel1, customProperties={}, snapshotId=null}, device_c8c2ea54-7116-47e0-ab6d-a6019b8b6e61device_c361fa4f-328a-40ff-95bf-bb5511367ed2device_fbd2ef8c-895a-44de-be3c-2d12c9735707device_3989012d-70d6-4125-a6a6-b456a8d6c03bdevice_1c86e7bc-ca6e-4a12-b5c2-e674fd6182cc=VmDevice {vmId=77195f8a-787f-4d55-a9d5-932bd5b29d4a, deviceId=1c86e7bc-ca6e-4a12-b5c2-e674fd6182cc, device=virtio-serial, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x05, function=0x0}, managed=false, plugged=true, readOnly=false, deviceAlias=virtio-serial0, customProperties={}, snapshotId=null}}
2014-08-08 17:43:32,332 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (DefaultQuartzScheduler_Worker-4) [5b72f572] FINISH, CreateVDSCommand, log id: 4afbd1c
2014-08-08 17:43:32,338 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (DefaultQuartzScheduler_Worker-4) [5b72f572] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 703d8622
2014-08-08 17:43:32,339 INFO [org.ovirt.engine.core.bll.RunVmCommand] (DefaultQuartzScheduler_Worker-4) [5b72f572] Lock freed to object EngineLock [exclusiveLocks= key: 77195f8a-787f-4d55-a9d5-932bd5b29d4a value: VM
, sharedLocks= ]
2014-08-08 17:43:32,344 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [5b72f572] Correlation ID: 5b72f572, Job ID: 10d06140-a522-475c-8f1d-fbfb3614d16a, Call Stack: null, Custom Event ID: -1, Message: VM v1 was restarted on Host 150
2014-08-08 17:43:32,391 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (DefaultQuartzScheduler_Worker-4) [58c9f55d] START, IsVmDuringInitiatingVDSCommand( vmId = 3248d236-eab3-4e3b-9fb1-325c6dd0fb67), log id: 542b3a51
2014-08-08 17:43:32,392 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (DefaultQuartzScheduler_Worker-4) [58c9f55d] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 542b3a51
2014-08-08 17:43:32,427 INFO [org.ovirt.engine.core.bll.RunVmCommand] (DefaultQuartzScheduler_Worker-4) [58c9f55d] Running command: RunVmCommand internal: true. Entities affected : ID: 3248d236-eab3-4e3b-9fb1-325c6dd0fb67 Type: VM
2014-08-08 17:43:32,522 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (DefaultQuartzScheduler_Worker-4) [58c9f55d] START, CreateVmVDSCommand(HostName = 150, HostId = 44379bb8-e87b-4e00-a18e-5df43922da82, vmId=3248d236-eab3-4e3b-9fb1-325c6dd0fb67, vm=VM [c6.5_1]), log id: 477642b8
2014-08-08 17:43:32,547 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (DefaultQuartzScheduler_Worker-4) [58c9f55d] START, CreateVDSCommand(HostName = 150, HostId = 44379bb8-e87b-4e00-a18e-5df43922da82, vmId=3248d236-eab3-4e3b-9fb1-325c6dd0fb67, vm=VM [c6.5_1]), log id: 6048754b[参考]
