Cisco UCS
Locked Out of Cisco UCS? How to Recover the Master Admin Password | Lazy Admin Blog
It’s the nightmare scenario: you need to make a critical service profile change, but the only admin password is lost or forgotten. Because Cisco UCS Manager doesn’t store passwords in a reversible format, you can’t “view” the old one. Instead, you must perform a password reset by power-cycling the Fabric Interconnects (FIs) and interrupting the boot sequence.
⚠️ Warning: This procedure requires a physical power cycle of the Fabric Interconnects. In a production environment, this will cause a temporary disruption in management connectivity and potentially data traffic if not handled correctly in a cluster.
Phase 1: The Pre-Flight Check
Before you pull the power cables, you need two pieces of information. If you still have read-only access or a lower-privilege account, gather these now:
- Identify the Roles: In a cluster, you must know which FI is Primary and which is Subordinate.
- Path: Equipment > Fabric Interconnects > [FI Name] > General > High Availability Details.
- Verify Firmware Versions: You must know the exact Kernel and System firmware versions currently running.
- Path: Equipment > Firmware Management > Installed Firmware.
Phase 2: Password Recovery (The Process)
Scenario A: Standalone Configuration
If you only have one Fabric Interconnect, the process is straightforward but requires downtime.
- Connect: Attach a console cable physically to the FI console port.
- Power Cycle: Turn the FI off and then back on.
- Interrupt Boot: As it boots, repeatedly press Ctrl+L or Ctrl+Shift+R until you see the
loader >prompt. - Boot Kernel: Load the kickstart/kernel image:
loader > boot /installables/switch/ucs-6100-k9-kickstart.x.x.x.gbin - Enter Config:
Fabric(boot)# config terminal - Reset Password:
Fabric(boot)(config)# admin-password YourNewPassword123 - Load System: Exit config mode and boot the system image:
Fabric(boot)# load /installables/switch/ucs-6100-k9-system.x.x.x.bin
Scenario B: Cluster Configuration (High Availability)
In a cluster, the order of operations is vital to ensure the database remains synchronized.
- Subordinate First: Power cycle the Subordinate FI and interrupt its boot to the
loader >prompt. Leave it there. - Primary Second: Power cycle the Primary FI and interrupt its boot to the
loader >prompt. - Reset on Primary: Follow the “Standalone” steps (4 through 7) on the Primary FI console.
- Bring up Subordinate: Once the Primary is back up and you can log into UCS Manager, go to the Subordinate console and boot its kernel and system images normally from the loader prompt.
Important Notes
- Clear Text: When you type the
admin-passwordcommand in the boot loader, the password displays in clear text on the screen. Ensure no one is shoulder-surfing! - Strong Passwords: UCS Manager requires at least one capital letter and one number.
- Console Access: This cannot be done via SSH. You must have physical or terminal server access to the console port.
#CiscoUCS #DataCenter #CiscoProphet #SysAdmin #Networking #ITTech #Cisco #UCSManager #LazyAdmin #Infrastructure
Mastering Memory: A Guide to the Cisco UCS B200 M3 Blade Server | Lazy Admin Blog
Optimizing a Cisco UCS B200 M3 blade server begins with proper memory configuration. In the enterprise world, an incorrectly seated DIMM or a mismatched channel doesn’t just lower performance—it can trigger a cascade of system errors and costly downtime.
🛠️ The Installation Procedure
Before you begin, ensure you are wearing an ESD (Electrostatic Discharge) wrist strap and that the blade is placed on an antistatic mat.
Step 1: Prepare the Slot
Locate the target DIMM slot and push the two white connector latches outward to the open position.
Step 2: Seat the DIMM
Align the notch on the bottom edge of the DIMM with the key in the slot.
- Precision is key: Press down evenly on both ends of the DIMM until the latches snap up and click into place.
- Warning: DIMMs are keyed. If it doesn’t seat with gentle pressure, check the alignment. Forcing a misaligned DIMM can permanently damage the motherboard or the module.
Step 3: Final Lock
Manually press the connector latches inward slightly to ensure they are fully seated and the DIMM is securely locked.
📐 Understanding Memory Architecture
The B200 M3 is a powerhouse, supporting up to 24 DIMM slots (12 per CPU). To maximize throughput, you must understand how these slots are mapped.
Channels and Slots
Each CPU manages four channels, with three DIMM slots per channel. Cisco uses a color-coding system to indicate the population order:
| Slot Number | Color | Order |
| Slot 0 | Blue | Populate First |
| Slot 1 | Black | Populate Second |
| Slot 2 | White/Beige | Populate Last |
Physical Mapping
- CPU 1 (Left): Manages Channels A, B, C, and D.
- CPU 2 (Right): Manages Channels E, F, G, and H.
[!IMPORTANT]
Single CPU Configurations: If only one CPU is installed, only the 12 slots associated with CPU 1 (left side) are functional. Memory installed in CPU 2 slots will not be recognized.
⚠️ Support and Compliance
- Third-Party Warning: Cisco does not support third-party memory. Using non-Cisco DIMMs can lead to “Inoperable” status in UCS Manager, hardware damage, or the denial of RMA requests.
- Verification: Always check the official Cisco Data Sheets for the latest supported DIMM capacities and speeds.
- Validation: After installation, boot into Cisco UCS Manager to verify that all DIMMs are discovered and show a “Healthy” status.
Courtesy: Cisco
Configuring Cisco NIC Teaming on UCS B200-M3
For Windows-based Cisco UCS B-Series blades, native teaming is often handled via the Cisco-specific driver contained in the UCS Windows Utilities ISO. Here is how to install and manage teams via the Command Line Interface (CLI).
Prerequisites
- Download the Windows Utilities ISO from Cisco.com.
- Choose either the B-Series Blade or C-Series Rack-Mount software bundle.
- Ensure you have Administrator privileges on the Windows target.
Phase 1: Installing the NIC Teaming Driver
The driver is installed using the enictool. You must point it to the directory containing the .inf files from the ISO.
- Open Command Prompt as Administrator.
- Run the following command:DOS
enictool -p "C:\path\to\drivers"Example:C:\> enictool -p "c:\temp"
Phase 2: Creating and Configuring the Team
Once the driver is active, you can group your logical interfaces into a team.
- Identify your connections: Use
ipconfigorncpa.cplto find the exact names (e.g., “Local Area Connection”). - Create the Team:DOS
enictool -c "Connection 1" "Connection 2" -m [mode]
Mode Reference Table
| Mode ID | Description | Best Use Case |
| 1 | Active-Backup | Basic redundancy; one link stays idle. |
| 2 | Active-Backup (Failback) | Redundancy; always reverts to the primary link when healthy. |
| 3 | Active-Active | Transmit Load Balancing; uses both links for outgoing traffic. |
| 4 | 802.3ad LACP | Link Aggregation; requires specific configuration on the Fabric Interconnect/Switch. |
Example (Active-Backup):
C:\> enictool -c "Local Area Connection" "Local Area Connection 2" -m 1
Phase 3: Management Commands
- To Delete a Team:
C:\> enictool -d "Local Area Connection" "Local Area Connection 2" - To View All Options:
C:\> enictool /?(Use this to fine-tune Load Balancing hash methods and advanced failover settings.)
#CiscoUCS #NICTeaming #SysAdmin #DataCenter #Networking #WindowsServer #TechTutorial #LazyAdmin #ServerAdmin #Infrastructure
Troubleshooting UCSM Login Errors After Java 7 Update 45
If you just updated your management workstation to Java 7 Update 45, you might find yourself locked out of Cisco UCS Manager (UCSM). This is a known issue caused by a change in how Java handles HTTP responses, which unfortunately breaks the communication handshake with certain UCSM versions.
The Symptoms
When attempting to launch the UCSM Java client, you will likely encounter one of these two frustrating login errors:
- Error 1:
Login Error: java.io.IOException: Invalid Http response - Error 2:
Login Error: java.io.IOException: Server returned HTTP response code: 400 for URL: http://x.x.x.x:443/nuova
The Cause: Cisco Bug CSCuj84421
This issue is officially tracked under Cisco Bug ID CSCuj84421. The security enhancements introduced in Java 7u45 changed the way the Java Runtime Environment (JRE) processes headers, leading to the “400 Bad Request” error when reaching out to the UCS virtual IP.
The Workaround: The Rollback
Until you can upgrade your UCSM firmware to a version that is compatible with newer Java releases, the most reliable solution is to downgrade your JRE.
- Uninstall Java 7 Update 45 from your system via the Control Panel.
- Download and Install an earlier version.
- Java 7 Update 25: Confirmed as the most stable version for this specific bug.
- Java 7 Update 40: Also reported to work by many in the Cisco community.
- Clear your Java Cache: After downgrading, go to the Java Control Panel > General > Temporary Internet Files > View, and delete the UCSM application entries to ensure a fresh launch.
Note: Always remember to disable “Check for Updates Automatically” in the Java Control Panel after rolling back, or you’ll find yourself back in the same position tomorrow morning!
#CiscoUCS #JavaErrors #SysAdmin #DataCenter #UCSM #TechSupport #ITTroubleshooting #CiscoNetworking #LazyAdmin #JavaUpdate
🏗️ CLI Command Hierarchy & Navigation
The CLI is organized like a file system. You move “down” into specific modes to manage objects and “up” to return to the global level.
- EXEC Mode (
#): The top-level mode. From here, you can access all other sub-modes. - Navigation Commands:
scope <object>: Moves into a sub-mode for an existing object (e.g.,scope chassis 1).enter <object>: Similar to scope, but used to enter or create an object’s mode.exit: Moves up one level in the hierarchy.top: Jumps immediately back to the EXEC mode prompt.
🛠️ Common Management Commands
| Target | Command | Purpose |
| Chassis | show chassis [inventory/status/psu] | View physical chassis health and components. |
| Servers | show server [inventory/cpu/memory/status] | Audit blade or rack-mount hardware specs. |
| Fabric | show fabric-interconnect [a/b] [inventory] | Check the state of your Fabric Interconnects. |
| Faults | show fault [detail/severity] | List active system alarms and errors. |
| Logs | show sel [chassis-id/blade-id] | View the System Event Log for specific hardware. |
💾 The Transactional Model (Commit Buffer)
Unlike many traditional CLIs, UCS Manager uses a transactional model. When you make a configuration change (like set or enable), the change is stored in a temporary buffer and is not live until you explicitly save it.
- Modify:
set addr 192.168.1.50 - Verify:
show configuration pending(Optional) - Apply:
commit-buffer - Discard:
discard-buffer(If you made a mistake)
#CiscoUCS #CommandLine #SysAdmin #DataCenter #Networking #Cisco #ITPro #LazyAdmin #TechTutorials #UCSM
Demystifying Cisco UCS Monitoring: Manager vs. Standalone C-Series
Whether you are managing a massive farm of B-Series blades or a handful of standalone C-Series rack servers, Cisco UCS provides a sophisticated, stateful monitoring architecture. Understanding how this “Queen Bee” and “Worker Bee” relationship works is the key to reducing alert fatigue and maintaining 100% uptime.
🏗️ The Architecture: DME and Application Gateways
The core of UCS monitoring relies on three primary components that translate raw hardware signals into human-readable data.
1. Data Management Engine (DME)
Think of the DME as the Queen Bee. It is the central brain that maintains the UCS XML Database. This database is the “Single Source of Truth” for your entire domain, housing inventory details, logical configurations (pools/policies), and current health states.
2. Application Gateways (AG)
The AGs are the Worker Bees. These are software agents that communicate directly with hardware endpoints (blades, chassis, I/O modules). They monitor health via the CIMC (Cisco Integrated Management Controller) and feed that data back to the DME in near real-time.
3. Northbound Interfaces
These are your outputs. You have Read-Only interfaces like SNMP and Syslog for external monitoring, and the XML API which is a Read-Write interface, allowing you to both monitor health and push configuration changes.
🚨 The Fault Lifecycle: Managing “State”
Cisco UCS doesn’t just send “fire and forget” alerts. It uses a stateful fault model. Faults are objects that transition through a lifecycle to prevent “flapping”—where a minor glitch sends dozens of emails in a minute.
- Active: The problem is occurring now.
- Soaking: The issue cleared quickly, but the system is waiting to see if it reoccurs before notifying you.
- Flapping: The fault is clearing and reoccurring in rapid succession.
- Cleared: The issue is fixed, but the record is retained briefly for your attention.
- Deleted: The fault is finally purged once the retention interval expires.
✅ Best Practices for the “Lazy Admin”
1. Filter out FSM Faults
In UCS Manager, Finite State Machine (FSM) faults are almost always transient. They occur during a task transition—like a server taking a bit too long to finish BIOS POST during a profile association.
The Rule: Focus your alerting on Major and Critical severities that are NOT of type FSM. This will eliminate about 80% of your monitoring “noise.”
2. Leverage Consistency
One of the best features of the UCS ecosystem is that Standalone C-Series and UCS Manager use the same MIBs and Fault IDs. If you have an NMS (Network Management System) set up for your blades, adding standalone rack servers is seamless because the data structure is identical.
3. Use Fault Suppression
Doing maintenance? Don’t let your monitoring system scream at you. Use the Fault Suppression feature (added in UCSM 2.1) to silence alerts on a specific blade or rack server while you are working on it.
4. The XML API Advantage
For standalone C-Series servers, the XML API is the preferred monitoring method. It supports Event Subscription, which proactively “pushes” alerts to your management tool rather than forcing the tool to “pull” or poll for data constantly.
CiscoUCS #SysAdmin #DataCenter #Networking #Cisco #ITPro #ServerMonitoring #LazyAdmin #Automation #TechTips
🏗️ The Architecture: How UCS Manager “Thinks”
For B-Series (blade) and integrated C-Series (rack) servers, monitoring is driven by a “Queen Bee and Worker Bee” relationship.
1. Data Management Engine (DME)
The DME is the brain of the system. It maintains the UCS XML database, which stores the current inventory, health, and configuration of every physical and logical component in your domain.
- Real-Time Only: By default, the DME only shows active faults. It does not store a historical log of everything that ever went wrong.
2. Application Gateway (AG)
The AGs are the “worker bees.” They communicate directly with endpoints (servers, chassis, I/O modules) to report status back to the DME.
- Server Monitoring: AGs monitor health via the CIMC (Cisco Integrated Management Controller) using IPMI and SEL logs.
3. Northbound Interfaces
These are the “outputs” that you, the administrator, actually interact with:
- SNMP & Syslog: Read-only interfaces used for external monitoring tools.
- XML API: A powerful “read-write” interface used for both monitoring and changing configurations.
🚨 Understanding Faults and Their Lifecycle
In Cisco UCS, a fault is a “stateful” object. It doesn’t just appear and disappear; it transitions through a specific lifecycle to prevent “alert fatigue” caused by temporary glitches.
The Fault Lifecycle
- Active: The condition occurs, and a fault is raised.
- Soaking: The condition clears quickly, but the system waits (the flap interval) to see if it comes back.
- Flapping: The fault is raised and cleared several times in rapid succession.
- Cleared: The issue is resolved, but the fault remains visible for a “retention interval” so you don’t miss it.
- Deleted: The fault is purged from the database.
✅ Best Practices for Monitoring
1. The “Severity” Rule
For UCS Manager, your monitoring tool should focus on faults with a severity of Critical or Major. Ignore “Info” or “Condition” alerts unless you are deep-diving into a specific issue.
2. Filter out “FSM” Faults
Finite State Machine (FSM) faults are usually transient. They often trigger during a task (like a BIOS POST during a service profile association) and resolve themselves on a second or third retry.
- Note: This only applies to UCS Manager. Standalone C-Series servers do not use FSM, so all their faults are usually relevant.
3. Use the XML API for C-Series
If you are managing standalone C-Series servers, the XML API is the gold standard. It supports Event Subscription, which pushes proactive alerts to you rather than making your tool “pull” data constantly.
📚 Essential Resource Links
Keep these bookmarked for when those cryptic SNMP OIDs start popping up in your logs:
#CiscoUCS #SysAdmin #DataCenter #Networking #Cisco #ITPro #ServerMonitoring #LazyAdmin #Virtualization #TechTutorials