Skip to content

Lottabytes

Adventures in Enterprise Systems Management and Automation

  • My Gitlab
Lottabytes

Category: AutoDeploy

Troubleshooting ESXi AutoDeploy

0
September 20, 2012

As you start to use ESXi AutoDeploy more you will run into occasions where you want to look behind the scenes and see what is going on. There are a couple ways to do so:

1. In vCenter click on the “Auto Deploy” button under Administration and hit “Download AutoDeploy Log Files”. This will give you a ZIP file of the logs on the AutoDeploy Server and provides lots of information on what is going on or what might be broken.

image

2. Go to https://<serverIP>:6501/vmw/rbd/host/

This will bring up a list of all the hosts that have called into the AutoDeploy Server. From there you can click on the host and get more information:

image

From this screen you can also go to the Get gPXE Configuration which shows you your boot information (including Host Profile and Image Profile being used,) or the Get boot.cfg which shows you what appears to be the actual download of the operating system files to the server.

AutoDeploy, ESXi, logs, troubleshooting

ESXi AutoDeploy–HP G7 Servers Hang on Boot

0
September 20, 2012

When I was first starting to use AutoDeploy I ran into an issue where my blades would start to load ESX and then hang just like the screen below shows. Inspection of the logs turned up very little data, including no errors. It was just like the server ceased to exist. Turns out if you go to HP’s site and download the new OneConnect Flash 4.1.450.7 and update the firmware on your Emulex NICs that will fix the issue. Also, something to note is once you update the firmware you will need to go to the VMware site and download the new driver VIBs for your host to work.

image

AutoDeploy, ESXi, G7, hangs on boot, HP

ESXi AutoDeploy Fails–No Image Profile Assigned

4
September 20, 2012

 

I was migrating blades today that initially had ESXi 5.0 installed to the local disk and wanted them to be installed via AutoDeploy and found an interesting issue. Upon rebooting the blades and changing the boot order I got a very interested error:

image

Upon further inspection I verified that the Deployment Rule was correct and that it was a member of a valid and active RuleSet and yet the blade would not boot via AutoDeploy. It kept telling me that “This host has been added to VC, but no Image Profile is associated with it.” I even created a new Deployment Rule and assigned based on the specific MAC Address to no avail… same error.

image

Turns out… you have to manually remove the ESXi host from vCenter or AutoDeploy will never work and you won’t ever see a more descriptive error. I was hoping that it would just update the host in vCenter but no dice…

AutoDeploy, but no Image Profile is associated with it., ESXi, Fails, This host has been added to VC

Using ESXi Autodeploy

0
September 19, 2012

I’m new to the ESXi Autodeploy bandwagon but I’ve got to say after using it for the first time I’m sold. If you are interested in what Autodeploy is a quick summary is that it allows you to dynamically install ESXi via PXE on servers and add them to your vSphere Infrastructure in an intelligent manner, all automatically.

For this real life example I have 2 Clusters of 7 HP BL490c G7 Blades that I need to dynamically provision with ESXi (complete with a custom NIC driver) and add them to the correct cluster based on IP ranges. Here is how I went about doing so.

Assumptions: I am assuming that you are not new to vSphere and that your environment already has a functioning vCenter Server (ver5), PowerCLI is installed, ESXi Coredump Collector and Autodeploy is installed. If you are missing any of these please follow the below link to get them installed before continuing: http://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.vsphere.install.doc_50%2FGUID-CAB84194-3D8E-45F0-ABF9-0277710C8F98.html

A couple other things are also needed:

1. You need to set the hardware clock on all your servers to have the correct date and time. If you don’t do this there is a chance that you will run into an issue where the blades will not add to the cluster and you will find that they are trying to use an evaluation license, even with bulk licensing correctly configured.

2. A TFTP Server must be installed. I used the free Solarwinds TFTP server which I installed on my vCenter Server.

a. Create a folder called TFTP-Root

b. Create a folder called ESX_Depots

c. Open you vCenter Client and on the home screen select the “Auto Deploy” option. From there click on the “Download TFTP Boot Zip” and extract the contents into the TFTP-Root folder you created earlier.

3. Create DHCP Reservations for all of your blade servers and give them the following options:

a. All normal networking options (003 – Router, 006 – DNS, 015 – DNS Domain Name, etc…)

b. Add Option 066 “Boot Server Host Name” and give it the value of your TFTP Server, in my case this is also my vCenter Server.

c. Add Option 067 “Bootfile Name” with the value of “undionly.kpxe.vmw-hardwired”. You will notice that this is a file that is now located in your TFTP-Root directory from the zip file you extracted there.

4. You should be able to boot one of your servers at this point and see a screen that informs you that it booted from PXE and no ESX image is assigned and your server will reboot in 5 minutes… So far, so good…

At this point we have got the infrastructure in place to support ESXi Autodeploy and we have to create the actual ESX images that we want to install. Also we need to create the Host Profile(s) that we want to use and add a custom networking driver because the one included by default does not work with this specific model and generation blade. Let’s get started:

1. Download ESX Image – At this point you need to download a couple things from the VMware website. The first is the .zip Depot files for the version of ESXi that you want to install. The second is a package called vSphere-HA-Depot or vmware-fdm that you will need if you are going to add your ESX hosts to a HA enabled cluster.

2. In my specific case the driver that is included with ESX for our NIC on the BL490c G7 blade has a bug that was fixed by HP; that means that I will need to download the additional driver as well. Most people will not need to do this step unless you are using HP G7 blades with the NIC issues.

Once you have downloaded the software depots that are needed you are ready to create the ESXi image. To do that we are going to use the versatile PowerCLI. (Hint: If you are ever wondering about the syntax of a command or what options you have there is a great list of commands and syntax located at http://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.vsphere.install.doc_50%2FGUID-16E1D78F-2466-4794-8D12-BE5EC7AA41D3.html)

Step 1: Adding software depots: We downloaded all these software depots (ESX, drivers and vmware-fdm) but now we need to make them accessible for use in the PowerCLI. To do that we add all of them via the syntax: Add-EsxSoftwareDepot –DepotUrl “c:esx_depot<filename>

image

Now we should be able to see all of our available software packages (VIBs) by issuing the command get-EsxSoftwarePackage:

clip_image004

On this screen you can see that we have 2 versions of the net-be2net, the newer one being from Emulex as well as a new file called vmware-fdm. Cool, we are on a roll….

Next we need to clone the original ESXi image so that we can make changes to it. First let me show you what the default looks like: Execute Get-EsxImageProfile and you should see the 2 entries for the version of ESX that was in your depot that you added. Now using New-EsxImageProfile –CloneProfile <original name> -Name <New Name> create a copy of the ESXi image that we will be editing with our driver and vmware HA Depot (vmware-fdm).

clip_image006

Now that we have our ESXi Image created we need to add the new driver and the vmware-fdm VIBs. Here’s how I did it:

1. For demonstration purposes I tried to remove the vmware-fdm and as you can see it failed because it is not included by default. Next I removed the Broadcom network driver from our Image. That one succeeded as expected.

clip_image008

2. Now I need to add the new network driver. In theory I could just specify the driver by name and it would install the new one because the version number is higher but just for grins I want to manually specify the entire driver name and version… You can see from the below screenshot that you can specify the short name or the name and version.

clip_image010

Now that we have this nice new shiny ESXi image with custom drivers, now what… Well, we need to create a Deployment Rule that has the logic that tells vCenter what servers get this image as well as what Host Profile to use but before we go there I need to throw out a couple hints.

Hint #1: On your first round you will want to create a deployment rule that does not use a host profile. Once that server is provisioned you will want to configure it and then use it as a template for creating your Host Profile that will be used to build all the other servers.

Hint #2: You will get a message that the password was not changed on the ESXi servers even if you create a host profile using a server as a template that has a valid root password. You need to manually edit your Host Profile after creating it to have the below information to get a new root password applied to your hosts as they come online.

clip_image012

Hint #3: You will most likely get a couple errors when you try and apply your host profile that say the Path Selection Policy needs changed and that the “device mpx.vmhba32:xx:xx:xx needs to be reset.” These errors are sometimes caused by virtual CDROMs or local storage and will cause your servers to fail compliance. This is easily avoided by disabling those options in the host profile if you so choose. A great article on doing so is located at http://bsmith9999.blogspot.com/2012/01/vmware-host-profiles-giving-errors.html

Hint #4: You can configure the ESXi Dump Collector via Host Profiles but for some reason I had to do it manually on my first host (the one that you create the host profile against) using the below command for it to work properly. Maybe just something goofy that I did, or maybe not… anyways here are the commands to do it manually:

Esxcli system coredump network set –interface-name vmk0 –server-ipv4 <ip of coredump server> –server-port 6500

Esxcli system coredump network set –enable true

And the location in the host profile:

clip_image014

Ok, now that you have created your host profile using the first host now it is time to refine your deployment rules. First off go ahead and delete your original deployment rule that did not include any host profiles or install patterns since it is no longer needed.

Remove-DeployRule –Name <Whatever you named it>

Now we create a new one that uses IPs to limit what servers get the ESXi image installed and assigns them the new Host Profile we just created.

clip_image016

As you can see from the above screenshot we do a couple things:

Create a new Deployment Rule called vCD_Cluster_4 that uses:

i. “Cluster_4_vDS Ready” Host Profile

ii. 5.0-GA ESXi Image Profile we created earlier

iii. Adds the servers that fall into the 10.25.96.29-35 to the “Cluster 4” Cluster in my vCenter. All servers that are outside of this range will not get this install.

Last of all we need to enable that Deployment Rule by issuing the below commands:

clip_image018

One last hint: If you have a Host Profile assigned to the cluster in vCenter that Host Profile takes precedence over the one that you specified in the Deployment Rules.

There you go, now you have a good idea on how to get your environment up and running via ESXi Autodeploy complete with HA VIBs and a custom driver. It’s really cool technology and now I don’t know why I’ll ever need an ISO of ESXi again…

AutoDeploy, Custom Drivers, ESXi, Host, Host Profiles, undionly.kpxe.vmw-hardwired, vCD, VIBs

vSphere AutoDeploy Fails – License Error

2
August 31, 2012

If you have a valid license assigned to your Datacenter using Bulk Licensing and yet when you boot a blade using AutoDeploy you get an error “License not Available to perform the operation. The Evaluation Mode license assigned to Host machine.sub.domain.com has expired” boot your servers into the BIOS and set the clock to the right time. Trust me… just do it…

Coming Soon – New post on configuring AutoDeploy complete with custom driver install and host profiles. Also a quick and easy guide to installing and configuring vCM. Gonna be fun.

AutoDeploy, Host Profiles, License not available to perform the operation, vCM, vSphere

About Me

My name is Caleb Stephenson, and I am a Sr. Cloud Reliability Engineer working on the Private Cloud Team at VMware and specializing in Systems Management and Automation. We make cloud computing happen at scale. If you have attended a VMworld since 2013 and taken a Hands-On-Labs or used HOL for another event, it is just one of the many things that we provide as a cloud provider.

I am a jack-of-all-trades and unfortunately, master at none. My various IT certifications over the years have included AWS Certified Solutions Architect – Associate,  VMware Certified Professional (VCP), Certified Scrum Product Owner (CSPO), ITIL-F, RHCSA (RHEL 7), MCSA 2003 and MCSE 2003. While I am an engineer, I am also the Product Owner for my team which means my career is a delicate balance of technical and managerial skills.

Everything on this blog should be considered mine personally and not representative of my employer. There may be stuff published here that is pertaining to VMware products or product issues/fixes or bits of code I wrote. My blog should be read just like any other blog and not as a representative of VMware in any way, shape or form.

Recent Posts

  • NFSv3 Usage and Audit Logging
  • EXEC useradd in Docker fills hard drive on host
  • Python Exception inside Try/Except Statement
  • Playing with Wavefront – Network Packet Loss
  • Playing with Wavefront – Missing Agents

Recent Comments

  • Caleb on Automatically Configure VMware Log Insight
  • Dan on Error Removing Host from vSphere
  • Caleb on Getting Fancy with Log Insight Alerting (aka. Monitoring DHCP pools via logs)
  • Caleb on Monitoring VMware vCenter Servers using HTTP Health checks
  • Karuna Yarlagadda on SSRS 2008 Domain User Issue

Archives

  • November 2018
  • August 2018
  • July 2018
  • December 2017
  • November 2017
  • October 2017
  • September 2017
  • June 2017
  • October 2016
  • September 2016
  • August 2016
  • July 2016
  • May 2016
  • March 2016
  • February 2016
  • December 2015
  • July 2015
  • July 2014
  • May 2014
  • March 2014
  • February 2014
  • November 2013
  • October 2013
  • August 2013
  • June 2013
  • February 2013
  • January 2013
  • December 2012
  • November 2012
  • October 2012
  • September 2012
  • August 2012
  • April 2012
  • March 2012
  • February 2012
  • November 2011
  • October 2011
  • August 2011
  • March 2011
  • February 2011
  • November 2010
  • October 2010
  • September 2010
  • July 2010
  • June 2010
  • May 2010
  • April 2010
  • March 2010
  • December 2009
  • July 2009
  • June 2009
  • May 2009
  • April 2009
  • March 2009
  • February 2009

Categories

  • .mp3
  • .vlcp
  • .wmv
  • 00000000000000d1
  • 0x8024402C
  • 2005
  • 2008
  • 29506
  • 5.5
  • 64bit VPN
  • 80072ee7
  • aag
  • Ac
  • Access is Denied
  • account
  • Acrobat
  • Active Directory
  • AD
  • Adobe
  • agent
  • agentless
  • alert
  • alwayson
  • analytics
  • ansible
  • API
  • Apple Mac XServer Cron Backup Windows File Server
  • APSB09-01
  • apt-get
  • ARP Poisoning
  • ASA
  • ASDM
  • audacity
  • Authentication
  • AutoDeploy
  • automation
  • availability group
  • Awesome
  • backup
  • BartPE
  • Bind
  • BitLocker
  • blinking folder with question mark
  • bluetooth
  • boot
  • bottleneck
  • bridge address-table
  • broken
  • bsod
  • Build
  • but no Image Profile is associated with it.
  • cannot display the page
  • Cannot open the Outlook Window
  • CDP
  • CentOS
  • cfapi
  • Change
  • chkdsk
  • Chuck Swindoll
  • CIM
  • Cisco
  • Cisco ASA 5510
  • cluster
  • cmd
  • Comparison
  • compliance
  • Configuration
  • Configuration Managenent
  • configure
  • Content Pack
  • converting
  • Crash
  • CSV
  • Custom Drivers
  • custom fact
  • database log
  • debugging
  • decrypting drive
  • deployment
  • detected corruption
  • DHCP
  • disable
  • dism
  • DN
  • DNS
  • Docker
  • does not have required permissions
  • Download
  • downtime.
  • Dr. Os Guinness
  • driver
  • drivers
  • DTS
  • ECM
  • Edge
  • error
  • Error: Enabling Active Directory failed
  • ESX
  • ESX Cluster
  • ESXi
  • ESXi 4
  • ethics
  • event viewer
  • exchange 2007
  • exhausted
  • Fails
  • FedEx
  • File DNS
  • find VM by MAC
  • finding rogue mac
  • firmware
  • fqdn
  • Freedom
  • FreeTDS
  • FTP
  • full
  • G7
  • get-vmhostadvancedconfiguration
  • GetDataBack for NTFS
  • Getting Started
  • Google Down
  • Government
  • guide
  • Hands On Labs
  • hangs
  • hangs on boot
  • Hardware
  • high CPU
  • History
  • Host
  • Host Profiles
  • HP
  • hung
  • hyperic
  • ID 57
  • identity source
  • IEESC
  • iLO driver
  • InetAddress Ping
  • InfluxDB
  • Install
  • Integrated
  • Integrity
  • IP-Pools
  • IPS
  • IPSec
  • iSCSI
  • Isolate
  • James
  • java
  • job
  • kb950772
  • kerberos
  • kernel
  • ldaps
  • License not available to perform the operation
  • linux
  • Linux Mint
  • log
  • Log Insight
  • log insight agent
  • Log Parsing
  • logging
  • Loginsight
  • logs
  • lust
  • Macbook
  • Make Availale Offline
  • McAfee
  • md5
  • Microsoft
  • mind
  • missing
  • missing public traffic
  • missing traffic
  • mntapi error: 176
  • module
  • Momentus XT
  • mon
  • monitoring
  • MSI
  • MSSQL
  • mssql. sql
  • multiple monitors
  • Nested 64bit
  • NetFN 0x36
  • NetGen
  • netio.sys
  • netios.sys
  • network
  • network adapter
  • new hire
  • ntbtlog.txt
  • Office12
  • Oops
  • Open Source
  • Openfiler
  • Orchestrator
  • Outage
  • Outlook 2007
  • P2V
  • password change
  • percent
  • performance issues
  • perl
  • plugins
  • pool
  • Postgres
  • PostgreSQL
  • PowerCLI
  • Powershell
  • PPTP
  • Process
  • Prometheus
  • proxy
  • Puppet
  • Puppet Master
  • purity
  • pymssql
  • python
  • reached target initrd default
  • reboot
  • recovery model
  • RedHat
  • removing bitlocker
  • replica
  • response traffic dropped
  • Review
  • RHEL
  • robocopy
  • routing
  • SCM
  • Script
  • Seagate
  • Security
  • selinux
  • serial
  • series
  • Serv-U
  • Server
  • server 2003
  • server 2008
  • Server 2012
  • services
  • set-vmhostadvancedconfiguration
  • sfc
  • sha-512
  • Sign-On and Discovery
  • simple
  • Socrates in the City
  • spn
  • SQL
  • SQL Management Studio Express
  • SS
  • ssl
  • sso
  • SSRS
  • SSRS 2008
  • svchost.exe
  • syslog
  • SYSTEM_THREAD_EXCEPTION_NOT_HANDLED
  • target
  • theology
  • This host has been added to VC
  • time
  • tongue
  • troubleshooting
  • tsdb
  • tsql
  • Uncategorized
  • undionly.kpxe.vmw-hardwired
  • unexpected
  • uninstall drivers
  • Update DNS
  • UPS
  • US
  • used
  • user
  • vC
  • vCD
  • vcenter operations manager
  • vcloud director
  • vCM
  • vCNS
  • vCO
  • vcops
  • vCSA
  • VDR
  • vFabric
  • VIBs
  • Virtual
  • vm
  • VMware
  • VMware Configuration Manager
  • VMware Data Recovery
  • VMware HA
  • VMware vCenter Configuration Manager
  • VMworld 2013
  • VMworld2013
  • vpxd_servicecfg
  • vRealize
  • vRLI
  • vs
  • vShield
  • vSphere
  • W32/Wecorl.a
  • Wavefront
  • Web Client
  • Wecorl.a
  • Windows
  • Windows 7
  • Windows Authentication
  • Windows cannot access the specified device
  • Windows Server 2003
  • Windows Update Error
  • Windows User Account Control (UAC) restrictions have been addressed
  • WinFF
  • winrm
  • wireshark
  • won't boot
  • wordpress
  • workaround
  • x64
  • XFCE
  • XP
  • You do not have the Backup and Restore Files user rights
  • zenoss

Meta

  • Log in
  • Entries RSS
  • Comments RSS
  • WordPress.org

WordPress Theme: Idealist

This site uses cookies: Find out more.