Querying NServiceBus' MSMQ messages using Powershell
I’ll show how you can use Powershell to turn the messages in NServiceBus’ audit and error queues (or any queue, really) into custom PSObjects that can be easily filtered and/or formatted, effectively providing a simple way to analyze NServiceBus communication.
Introduction
We’re using NServiceBus to integrate quite a few of the applications used by Info Support internally, and at one point, I needed to be able to look under the hood to see how some of the messages flowed between these applications.
Because Powershell can be used to access the MSMQ queues and has excellent XML and querying/filtering support, I figured it could go a long way in making this information more accessible – and this is what I’m going to demonstrate here.
I’m using NServiceBus 4.7 with MSMQ transport and XML message serialization, and Powershell 4.
Reading Messages from a queue
In order to be able to use the .NET MessageQueue, we’ll need to load the System.Messaging.dll into our Powershell session:
[reflection.assembly]::LoadWithPartialName("System.Messaging")
Next. we’ll create the MessageQueue object that represents the local “audit” queue in which NServiceBus stores a copy of all processed messages.
$queueName = ".private$audit" $queue = new-object System.Messaging.MessageQueue($queueName) # To access a private queue remotely, use the following syntax: # $queueName = "FormatName:DIRECT=OS:<servername>private$audit"
Here’s a caveat: By default, when retrieving Messages from a queue only some Message properties are set; this is determined by the MessageReadPropertyFilter. NServiceBus stores all its metadata in the Message.Extension
property, so make sure that’s retrieved as part of the message:
$queue.MessageReadPropertyFilter.Extension = $true
Now we can retrieve all Messages in the queue:
# Retrieve all messages at once (could use a lot of memory) $messages = $queue.GetAllMessages() # Or, use this to get the messages in a more stream-like fashion: # $messages = $queue.GetMessageEnumerator2()
Inspecting the message contents
If you’re using NServiceBus with the XML serializer, the Message.Body
is best read into an XmlDocument
. For example, to display the contents of the first message:
$message = $messages[0] $bodyXml = new-object System.Xml.XmlDocument $bodyXml.PreserveWhitespace = $true $bodyXml.Load($message.BodyStream) $bodyXml.OuterXml
Another interesting piece is the NServiceBus metadata, which is stored in the Extension
property as XML:
$ms = new-object System.IO.MemoryStream(,[byte[]]$message.Extension) $extensionXml = new-object System.Xml.XmlDocument $extensionXml.PreserveWhitespace = $true $extensionXml.Load($ms) $ms.Close() $extensionXml.OuterXml
It contains a collection of key/value pairs with all kinds of useful information (and for messages in the error queue, this also includes details about the exception that occured):
<?xml version="1.0"?> <ArrayOfHeaderInfo xmlns_xsd="http://www.w3.org/2001/XMLSchema" xmlns_xsi="http://www.w3.org/2001/XMLSchema-instance"> <HeaderInfo> <Key>NServiceBus.MessageId</Key> <Value>76fd8cf2-7321-4e91-982a-a49600bfdd19</Value> </HeaderInfo> <HeaderInfo> <Key>NServiceBus.CorrelationId</Key> <Value>12685260-0f3f-4947-b481-a49600bfdd2c</Value> </HeaderInfo> ... <HeaderInfo> <Key>NServiceBus.MessageIntent</Key> <Value>Publish</Value> </HeaderInfo> <HeaderInfo> <Key>NServiceBus.Version</Key> <Value>4.7.5</Value> </HeaderInfo> <HeaderInfo> <Key>NServiceBus.TimeSent</Key> <Value>2015-05-12 09:38:33:255622 Z</Value> </HeaderInfo> ...
Now that you know how to inspect the NServiceBus message details for individual MSMQ messages, lets try and make this information more accessible so that we’re better able to filter and display multiple messages.
Creating custom PSObjects and extracting XML data
We’re going to create a Powershell function that turns a Message into a custom PSObject
with all useful message and metadata information as easily accessible properties. This function takes its input from the Powershell pipeline…
function ConvertTo-NServiceBusMessage() { param( [Parameter(ValueFromPipeline = $true, Mandatory = $true)] [System.Messaging.Message]$message ) Process { # Create a new PSObject with a Message property for the original MSMQ message. $result = New-Object PSObject $result | Add-Member -MemberType NoteProperty -Name "Message" -Value $message return $result } }
…so that we can pipe the Message
s to it directly from the queue:
$queue.GetAllMessages() | ConvertTo-NServiceBusMessage
Right now each PSObject
only contains a Message
property for the original message, which isn’t particularly useful. So, let’s add BodyXml
and MetadataXml
properties to it:
Process { # ... # Our NServiceBus messages are serialized to xml, provide it as an XmlDocument property named BodyXml. # Note that NServiceBus subscription requests don't have a body. $bodyXml = new-object System.Xml.XmlDocument if($message.BodyStream.Length -gt 0) { $bodyXml.Load($message.BodyStream) } $result | Add-Member -MemberType NoteProperty -Name "BodyXml" -Value $bodyXml # Most of NServiceBus' metadata is stored as xml in the Extension part of a message. $metadataXml = new-object System.Xml.XmlDocument if(($message.Extension.Length) -gt 0) { $ms = new-object System.IO.MemoryStream(,[byte[]]$message.Extension) $metadataXml.Load($ms) $ms.Close() } $result | Add-Member -MemberType NoteProperty -Name "MetadataXml" -Value $metadataXml return $result }
Even better, because the NServiceBus metadata is already organized as key/value pairs, we can add a separate NoteProperty
for each entry as well:
Process { # ... # The metadata is made up from key/value pairs in a series of HeaderInfo elements. Add a property to the $result # object for each of these key/value pairs. $metadataXml.SelectNodes("//HeaderInfo") | ForEach-Object { $key = $_["Key"].InnerText $value = $_["Value"].InnerText # Some properties are known to represent a DateTime; parse these as such. if($key -eq "NServiceBus.TimeSent" -or $key -eq "NServiceBus.TimeOfFailure") { $value = [System.DateTime]::ParseExact($value, "yyyy-MM-dd HH:mm:ss:ffffff Z", [System.Globalization.CultureInfo]::InvariantCulture) } $result | Add-Member -MemberType NoteProperty -Name $key -Value $value } return $result }
Obviously, we can also extract information from the mesage body and add that as separate properties. We’ll have to fiddle with namespaces though, because the NServiceBus XML serializer places the serialized message data in an xml default namespace that is derived from the message’s .NET namespace.
For example, the .NET type MyCompany.Acme.Events.ContactUpdated
would be serialized as:
<?xml version="1.0"?> <Messages xmlns_xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns_xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://tempuri.net/MyCompany.Acme.Events"> <ContactUpdated> <Contact> <Id>e2cab171-cf5c-dd11-9108-00155d01eb04</Id> <FullName>Petersen, Bob</FullName> ... </Contact> <ContactUpdated> </Messages>
In order to query this xml with XPath, we need a XmlNamespaceManager
because XPath 1.0 doesn’t provide a way to access elements in the default namespace without a prefix. So instead, we’ll have the prefix “ns
” correspond to the document’s default namespace:
$nsmgr = new-object System.Xml.XmlNamespaceManager($bodyXml.NameTable) $nsmgr.AddNamespace("ns", $bodyXml.DocumentElement.NamespaceURI)
We can now access the elements in the default namespace, for example to extract the message type with (don’t forget to specify the namespace manager):
# The Messages' first child element defines the NServiceBus message name. $messageType = $bodyXml.SelectSingleNode("/ns:Messages/*[1]", $nsmgr).LocalName Add-Member -InputObject $message -MemberType NoteProperty -Name "ESB.MessageType" -Value $messageType
Or, to extract some identifying information such as a contact or company name:
$targetName = $bodyXml.SelectSingleNode("//ns:Contact/ns:FullName | //ns:Company/ns:Name", $nsmgr).InnerText Add-Member -InputObject $message -MemberType NoteProperty -Name "ESB.TargetName" -Value $targetName
Note that Powershell also supports dot notation for XmlElement
s, where child elements and attributes can be accessed as if they were properties. If you know what you’re looking for (i.e. when you don’t need the XPath “//
” operator) this is generally more readable than using XPath.
For example, determining the contact’s full name could also be done like this:
$targetName = $bodyXml.Messages.ContactUpdated.Contact.FullName
For an overview of how to handle XML with Powershell, check out this excellent article: PowerShell Data Basics: XML
Presenting the results
If you run Get-Member
against the result…
$queue.GetAllMessages() | ConvertTo-NServiceBusMessage | Get-Member
…you’ll find that there is now indeed a property for everything we’ve added, including all metadata properties:
TypeName: System.Management.Automation.PSCustomObject Name MemberType Definition ---- ---------- ---------- ... $.diagnostics.originating.hostid NoteProperty System.String $.diagnostics.originating.hostid=f00621fce72168618b86cec9be3416ed BodyXml NoteProperty System.Xml.XmlDocument BodyXml=#document ESB.MessageType NoteProperty System.String ESB.MessageType=ContactUpdated ESB.TargetName NoteProperty System.String ESB.TargetName=Petersen, Bob Message NoteProperty System.Messaging.Message Message=System.Messaging.Message MetadataXml NoteProperty System.Xml.XmlDocument MetadataXml=#document NServiceBus.ControlMessage NoteProperty System.String NServiceBus.ControlMessage=True NServiceBus.CorrelationId NoteProperty System.String NServiceBus.CorrelationId=7c8cf79e-31bc-4a47-9f7b-a42900e8d9ee NServiceBus.MessageId NoteProperty System.String NServiceBus.MessageId=7c8cf79e-31bc-4a47-9f7b-a42900e8d9ee NServiceBus.MessageIntent NoteProperty System.String NServiceBus.MessageIntent=Subscribe NServiceBus.OriginatingAddress NoteProperty System.String NServiceBus.OriginatingAddress=AcmeAdapter@TSTSERVER ...
You can now use these properties to filter the Messages with and to display them in a grid (note that a lot of the property names have a ‘.’ in them, and so need to be wrapped in quotes when using them):
$queue.GetAllMessages() | ConvertTo-NServiceBusMessage | Where-Object { $_.'NServiceBus.MessageIntent' -eq 'Publish' } | Format-Table -AutoSize -Property @('NServiceBus.TimeSent', 'NServiceBus.ProcessingEndpoint', 'NServiceBus.EnclosedMessageTypes')
Or, if you want full control over how the results are displayed:
$format = @( @{Label="MessageId"; Width=40; Expression={$_.'NServiceBus.MessageId'}}, @{Label="Sent"; Width=25; Expression={$_.'NServiceBus.TimeSent'}}, @{Label="From"; Width=10; Expression={$_.'NServiceBus.OriginatingEndpoint' -replace "MyCompany.(w+).ServiceBusAdapter", '$1'}}, @{Label="To"; Width=10; Expression={$_.'NServiceBus.ProcessingEndpoint' -replace "MyCompany.(w+).ServiceBusAdapter", '$1'}}, @{Label="Intent"; Width=10; Expression={$_.'NServiceBus.MessageIntent'}}, @{Label="Message type"; Width=20; Expression={$_.'ESB.MessageType'}}, @{Label="Target name"; Width=20; Expression={$_.'ESB.TargetName'}}, @{Label="Error"; Width=140; Expression={$_.'NServiceBus.ExceptionInfo.Message'}}) $queue.GetAllMessages() | ConvertTo-NServiceBusMessage | Where-Object { $_.'ESB.MessageType' -like 'Contact*' } | Format-Table $format
Produces the following:
MessageId Sent From To Intent Message type Target name Error --------- ---- ---- -- ------ ------------ ----------- ----- fd3ea843-941c-4ebc-9cbd-a43a00ea6982 2/9/2015 2:13:28 PM Crm Sales Publish ContactUpdated Anderson, Alice 4e9dcf54-a8b9-43f5-9298-a45600c5a40f 3/9/2015 11:59:35 AM Crm Sales Publish ContactUpdated Barton, Bob c6e9fc8b-755f-45f2-bcf5-a497009eabbd 5/13/2015 9:37:42 AM Crm Shipping Publish ContactCreated Cross, Carol ...
Conclusion
As you can see, the combination of Powershell, NServiceBus, MSMQ and XML is a very powerful one. I recommend keeping this in your bag of tricks for the next time you need to figure out how a particular set of NServiceBus messages were sent and/or what went wrong.