This project has moved. For the latest updates, please go here.

Option to have client.Get return null instead of throwing exception if row was not found?

Jan 22, 2013 at 12:57 PM

With the official table storage client, it's possible to have the client to return null instead of throwing an exception when a row isn't found. For example with the older Azure Sdk:

TableServiceContext.IgnoreResourceNotFoundException = true

This may come in handy, as some people prefer to handle the nulls in code rather than wrapping things around try-catch blocks.

I wonder, does Lucifure Stash support this feature? If not, is it possible to add this? I went through the code (using Intellisense, as I don't get F# :) and I think this option could fit nicely in the StashClientOptions.

Coordinator
Jan 22, 2013 at 7:00 PM

Currently this is not supported.
I will consider adding this feature.
Thanks for the request! 

Coordinator
Jan 27, 2013 at 4:22 AM

Miksu, I added this feature such that on the Get(...) method exception throwing can be disabled using an option as described by you.

Jan 30, 2013 at 12:07 PM

Thanks! The new feature works nicely.

One question (which really doesn't fit into this topic so please let me know if you want me to open a new thread for this): Does the Lucifure Stash implement a retry policy for dealing with Azure's transient errors?

Coordinator
Jan 31, 2013 at 4:19 AM
RetryPolicy is available as a property on StashClientOptions and defaults to RetryPolicies.GetExponential().
Feb 1, 2013 at 4:58 AM
Great, thanks for the fast reply.

We did a batch run where we moved a little over 10 million rows from a storage account to an another. We had some existing code to read the messages but we used Lucifure Stash to insert the messages. We executed the following code in our loop:
 var existingDaily = newStorage.Get(dailyRow.PartitionKey, dailyRow.RowKey);

 if (existingDaily != null)
 {
     continue;
 }

 newStorage.Insert(dailyRow);
Where newStorage is Lucifure Stash client. So we first check if the row already exists (as we already had moved some of the messages) and then inserted the row.

Mainly this worked just great but we received around 20-30 exceptions in either the Get or the Insert. All the exceptions seem to be related to timeouts. I wonder if it's possible that if a timeout happens, the operation isn't retried? I think (but I'm not sure) that all these failed operations would have succeeded with a retry.

Here are the exceptions which we encountered:

1:
CodeSuperior.Lucifure.StashRuntimeException: Unexpected runtime error. Please examine the InnerException for more information. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.ConnectStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.IO.StreamReader.ReadBuffer()
at System.IO.StreamReader.ReadToEnd()
at CodeSuperior.Lucifure.RestRequestBuilder.GetResponse(HttpWebRequest request)
at <StartupCode$Lucifure-Stash>.$StashClient.makeRequest'@131.Invoke(HttpWebRequest request, String content, Int32 attempt)
at CodeSuperior.Lucifure.StashClient1.makeRequest(HttpWebRequest request, String content)
at CodeSuperior.Lucifure.StashClient
1.insert(a instance)
at <StartupCode$Lucifure-Stash>.$StashClient.Insert@689.Invoke(Unit unitVar0)
at CodeSuperior.Lucifure.Exp.wrap[a](FSharpFunc2 f)
--- End of inner exception stack trace ---
at Microsoft.FSharp.Core.Operators.Raise[T](Exception exn)
at CodeSuperior.Lucifure.Msg.RaiseInner[a](StashMessage msg, Exception ex)
at CodeSuperior.Lucifure.Exp.wrap[a](FSharpFunc
2 f)
at CodeSuperior.Lucifure.StashClient`1.Insert(a instance)
at EventsBatchConversion.Program.<>c__DisplayClass5.<Main>b__3(Guid id)
2:
CodeSuperior.Lucifure.StashRuntimeException: Unexpected runtime error. Please examine the InnerException for more information. ---> System.Net.WebException: The request was aborted: The request was canceled.
at System.Net.ConnectStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.IO.StreamReader.ReadBuffer()
at System.IO.StreamReader.ReadToEnd()
at CodeSuperior.Lucifure.RestRequestBuilder.GetResponse(HttpWebRequest request)
at <StartupCode$Lucifure-Stash>.$StashClient.makeRequest'@131.Invoke(HttpWebRequest request, String content, Int32 attempt)
at CodeSuperior.Lucifure.StashClient1.makeRequest(HttpWebRequest request, String content)
at CodeSuperior.Lucifure.StashClient
1.insert(a instance)
at <StartupCode$Lucifure-Stash>.$StashClient.Insert@689.Invoke(Unit unitVar0)
at CodeSuperior.Lucifure.Exp.wrap[a](FSharpFunc2 f)
--- End of inner exception stack trace ---
at Microsoft.FSharp.Core.Operators.Raise[T](Exception exn)
at CodeSuperior.Lucifure.Msg.RaiseInner[a](StashMessage msg, Exception ex)
at CodeSuperior.Lucifure.Exp.wrap[a](FSharpFunc
2 f)
at CodeSuperior.Lucifure.StashClient`1.Insert(a instance)
at EventsBatchConversion.Program.<>c__DisplayClass5.<Main>b__3(Guid id)
3:
CodeSuperior.Lucifure.StashRuntimeException: Unexpected runtime error. Please examine the InnerException for more information. ---> System.Net.WebException: The request was aborted: The request was canceled.
at System.Net.ConnectStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.IO.StreamReader.ReadBuffer()
at System.IO.StreamReader.ReadToEnd()
at CodeSuperior.Lucifure.StashClient1.getResponseStream(WebResponse response)
at <StartupCode$Lucifure-Stash>.$StashClient.makeRequest'@131.Invoke(HttpWebRequest request, String content, Int32 attempt)
at CodeSuperior.Lucifure.StashClient
1.makeRequest(HttpWebRequest request, String content)
at CodeSuperior.Lucifure.StashClient1.get(String partitionKey, String rowKey)
at <StartupCode$Lucifure-Stash>.$StashClient.Get@849.Invoke(Unit unitVar0)
at CodeSuperior.Lucifure.Exp.wrapIgnoreResourceNotFoundException[a](FSharpFunc
2 f)
--- End of inner exception stack trace ---
at Microsoft.FSharp.Core.Operators.Raise[T](Exception exn)
at CodeSuperior.Lucifure.Msg.RaiseInner[a](StashMessage msg, Exception ex)
at CodeSuperior.Lucifure.Exp.wrapIgnoreResourceNotFoundException[a](FSharpFunc2 f)
at <StartupCode$Lucifure-Stash>.$StashClient.-ctor@86-11.Invoke(FSharpFunc
2 f)
at CodeSuperior.Lucifure.StashClient`1.Get(String partitionKey, String rowKey)
at EventsBatchConversion.Program.<>c__DisplayClass5.<Main>b__3(Guid id)
4:
CodeSuperior.Lucifure.StashRuntimeException: Unexpected runtime error. Please examine the InnerException for more information. ---> System.Net.WebException: The operation has timed out
at System.Net.HttpWebRequest.GetResponse()
at CodeSuperior.Lucifure.RestRequestBuilder.GetResponse(HttpWebRequest request)
at <StartupCode$Lucifure-Stash>.$StashClient.makeRequest'@131.Invoke(HttpWebRequest request, String content, Int32 attempt)
at CodeSuperior.Lucifure.StashClient1.makeRequest(HttpWebRequest request, String content)
at CodeSuperior.Lucifure.StashClient
1.get(String partitionKey, String rowKey)
at <StartupCode$Lucifure-Stash>.$StashClient.Get@849.Invoke(Unit unitVar0)
at CodeSuperior.Lucifure.Exp.wrapIgnoreResourceNotFoundException[a](FSharpFunc2 f)
--- End of inner exception stack trace ---
at Microsoft.FSharp.Core.Operators.Raise[T](Exception exn)
at CodeSuperior.Lucifure.Msg.RaiseInner[a](StashMessage msg, Exception ex)
at CodeSuperior.Lucifure.Exp.wrapIgnoreResourceNotFoundException[a](FSharpFunc
2 f)
at <StartupCode$Lucifure-Stash>.$StashClient.-ctor@86-11.Invoke(FSharpFunc`2 f)
at CodeSuperior.Lucifure.StashClient`1.Get(String partitionKey, String rowKey)
at EventsBatchConversion.Program.<>c__DisplayClass5.<Main>b__3(Guid id)
Coordinator
Feb 3, 2013 at 5:34 AM
I am delighted that you could use Stash to move more than 10 million rows.

Although the retry policy is in place, there were a couple defects which could have caused less retries and short time periods between the retries. I believe that I have fixed all the issues.

In the Lucifure.Stash.Tutorial project, 99.Helper.cs file, I have added a method 'AddRetryInterceptor' which shows how an retry interceptor can be plugged into the StashClientOptions.

If you do another large run, I would appreciate plugging in the retry interception and logging how many actual retries took places and what was the success of the retries actually resolving the problem.

Again, I appreciate your effort in putting Lucifure Stash to good use.

P.S.

In future please create a new issue, so we can track it more easily. If not an issue, do create a new discussing thread.
Feb 4, 2013 at 5:11 PM
Thank you for the reply and for the code changes. We're planning to do a new batch run later this week and I'm sure to use the latest version and to log the retries.

Thanks again.