The Lost Brain
Thoughts, ideas and other such things of my wandering brain.

Dealing with XML normalization in ASMX Web Services

May 11, 2010 22:34 by TheLostBrain
In working with ASMX web services it's likely that you'll eventually run into the 'normalization' that gets applied to string data of web method parameters and return values.

This is most often experienced as .Net's enforcement of the XML spec for 'End of Line handling'...which basically states that all EOL variations are to be 'normalized' down to a single 'os independant' representation... which in this case is simply LF (\n...0x0A...10).

So in VB terms... all vbCRLF's will simply become vbLF.

Now here's the kicker - This normalization doesn't occur durring the serialization phase...but durring the DEserialization phase! So, this basically means that when we populate a string property and shoot it across the wire... the data is NOT normalized (it's shot out with those vbCRLFs still happily in place)! However, when the xml reaches the receiving side it IS normalized as part of the deserialization phase (and thus our vbCRs are stripped away). Is it just me or does this seem like it's totally backwards? It seems as though .Net would want to enforce the XML spec prior to sending the xml out over the wire.

So, why does all this matter? Well let's say you have simple client app where a user can fill out a multi-line textbox and then submit it to a web service which then writes it to a table in a database. Well, from what we just learned ... after the text is submitted it will be normalized and the string will then be stored in the database with its CRs stripped out. Now, this *might* be acceptable if the .net controls (textboxes, etc.) respected LFs by themselves as newlines...but they don't... they just show them as square-looking charecters. So now when you read the data back out and display it in a standard control the text will not be formatted correctly.


Well... after what felt like an endless amount of Googling... an elegant solution was NEVER found! The most common solution offered was to simply do a .replace(vbLf, vbCrLf) on EVERY individual proprty that was to house multi-line data. Not only is this not elegant but it also requires continual maintenance as new properties are added/removed, etc... And this REALLY didn't fit in line with the project that this solution was needed for as the types being exposed were auto-generated 'LINQ to SQL' types and the thought of having to manually maintain code for auto-generated properties just seemed completely counter-productived (especially since the table definitons were still changing quite a bit at that point).


In trying to come up with a reusable, automated, drop-in solution... quite a bit of time was invested in learning about soap plugins... but alas.. unfortunately even this didn't provide a viable solution.


In the end the solution relized simply builds upon the most popular one that's currently offered. However, instead of having to manually implement and maintain a '.replace(vbLf, vbCrLf) ' on every string property... we instead use a combination of reflection and anonymous types to offer a more automated answer. Using this solution one only needs to call a single method passing the the type itself in as the parameter. The method then iterates just the string properties of the type...and then only runs the replacement code on those properties that contain LFs. The code for this can be found below in the 'Web Service Side' section.


Now, because the normalization only occurs during DEserialization, when that same data is pulled back out of the DB (or wherever it's being stored) as part of another web method call... it will be sent out UNnormalized... and guess what... the standard web services proxy code that is auto-generated when setting up a web reference to a web service expects the data to be normalized... and it will throw an exception if it's not! Again, just sounds completely backwards to me...

So in order to get around this we can either implement a two phase solution where we:
1) Setup a Soap Extension that essentially re-normalizes the data right before it's sent across the wire (SoapMessageStage.AfterSerialize)...and then...
2) Use the same method we discussed above...but now also implement it on the Client Side for all incoming deserialized types.
OR
We can take advantage of a single step solution where we simply override the 'GetReaderForMessage' method of the autogenerated web service proxy class where we simply turn normalization off.
To do this you would simply create a new code file in the client project and then dump in the code from the 'Client Side' section below....being sure to make updates where neccessary.

It's important to note that while the second approach has many advantages, there is one caveat. Because we are not helping .net to re-normalize the data (as discused in solution 1, step 1 above) it is not meeting the xml spec for EOL handling...and can cause issues for other developers if you intend to make your web service publicly consumable. In our case however we were dealing with a proprietary web service that would not be consumed outside our codebase so considering it is a much more automated means to getting the job done coupled with the fact that it's less expensive in terms of cpu cycles... approach 2 was the definitely the better solution for our particular situation.



------Web Service Side BEGIN------

Private Function FixEolNormalization(Of t)(ByVal input As t) As t

Dim propertyInfo() As System.Reflection.PropertyInfo = GetType(t).GetProperties
Dim stringPropertyValue As String

For Each info In propertyInfo

If info.PropertyType.ToString = "System.String" Then

stringPropertyValue = GetProperty(input, info.Name, "")

If stringPropertyValue.Contains(vbLf) Then

SetProperty(input, info.Name, ReplaceCrWithCrLf(stringPropertyValue))

End If

End If

Next

Return input

End Function

Private Function SetProperty(ByVal obj As Object, ByVal propertyName As String, ByVal val As Object) As Boolean
Try
' get a reference to the PropertyInfo, exit if no property with that name
Dim pi As System.Reflection.PropertyInfo = obj.GetType().GetProperty(propertyName) 
If pi Is Nothing Then Return False
 
' convert the value to the expected type
val = Convert.ChangeType(val, pi.PropertyType)
 
' attempt the assignment
pi.SetValue(obj, val, Nothing)
Return True
Catch
Return False
End Try
End Function

Private Function GetProperty(ByVal obj As Object, ByVal propertyName As String, ByVal defaultValue As Object) As Object
Try
Dim pi As System.Reflection.PropertyInfo = obj.GetType().GetProperty(propertyName)
If Not pi Is Nothing Then
Return pi.GetValue(obj, Nothing)
End If
Catch
End Try

' if property doesn't exist or throws
Return defaultValue

End Function

 

Private Function ReplaceCrWithCrLf(ByVal str As String) As String

str = str.Replace(vbCrLf, vbLf)
str = str.Replace(vbCr, vbLf)
str = str.Replace(vbLf, vbCrLf)

Return str

End Function

------Web Service Side END------






------Client Side BEGIN-----

Namespace QuoteValetWebService
Partial Public Class QuoteValetWebService
Inherits System.Web.Services.Protocols.SoapHttpClientProtocol

Protected Overrides Function GetReaderForMessage(ByVal message As System.Web.Services.Protocols.SoapClientMessage, ByVal bufferSize As Integer) As System.Xml.XmlReader

'Turn of normalization as we read back in so we can accept VbCrLfs
Dim reader As System.Xml.XmlTextReader = DirectCast(MyBase.GetReaderForMessage(message, bufferSize), System.Xml.XmlTextReader)
reader.Normalization = False
Return reader

End Function

End Class
End Namespace

------Client Side END-----

Make your flash drive bootable! (DOS boot with access to entire flash drive)

January 4, 2008 13:19 by TheLostBrain

So you say want to make your flash drive DOS bootable so you can do all kinds of cool stuff like easily flashing your bios, etc.?
...And you want to have access to all of the space available on that flash drive once booted into DOS too?!

Ok... Easy! 

 

To get started you need just two things:

1) HP USB Disk Storage Format Tool (Works with any brand flash drive!)
(This is an older version that seems to be more "compatible"... it's worked very well for me under both XP and Vista!)
HPFLASH1.EXE (1.98 mb)

2) DOS System files (These are needed by the HP utility to make the flash drive bootable):
SystemFiles.zip (151.42 kb)

 

Please keep in mind that this process will FORMAT your flash drive and any data on it will be lost! So be sure to backup your flash drive before starting this process! All the usual "I will not be held responsible” jargon applies here! So in other words… If your machine dies or you loose data, etc in  relation to any of this I won't be held responsible.

 
Now then...


1) Start by installing the HP USB Disk Storage Format Tool (HPFLASH1.EXE)

2) Now open the SystemFiles.zip file and extract the contents into a seperate folder

3) Launch the utility and set the options using the screenshot below as a guide.
(Be sure to select FAT for the "File system" dropdown... I've not had any good luck at all with FAT32 or any of the others.)
(Scratch that... FAT32 has been working just fine and you'll need that for flash drives larger than 2GB, etc.)
(The "DOS system files" it's refering to are the ones you extracted in step 2, above.)




4) Click Start

5) You'll get a warning letting you know that your about loose all data on the flash drive... As long as you backed up your data go ahead and click "Yes".
(If you got a warning indicating the device was in use it's most likely because you have an explorer window opened to the flash drive directory, etc. Just close out any other running apps and try again.)


That's it! At this point you should have a flash drive that you can boot off of! :)  Keep in mind that you may need to tweek the boot options in your bios, etc. to boot from the flash drive but you should be good to go otherwise.

 

Note: Depending on how your Windows Folder Options are configured you may or may not see any files at all on your flash drive after the HP Utility is done. Rest assured they are there...they're just hidden. To confirm their existance you could either set your folder options to show hidden and system files or just simply drop down to a command prompt and run attrib against the flash drive letter as shown in the following screenshot:




This flash drive's ready to go! You can now copy all of your files back on to it... Just be sure to never delete those three system files and it should always be ready to boot! ;)


Questions / Comments? Leave 'em here. :)
Tags:
Categories: Tech
Actions: E-mail | Permalink | Comments (0) | Comment RSSRSS comment feed