I am running a thrift service on an EC2 instance AWS that is written in C# and running under Mono 3.2 (using the latest AWS .net SDK 1.5.28.2). The service hits dynamodb, and I am constantly getting low level networking errors when talking to dynamo. My current workaround is to just catch any AmazonServiceException and try again, which seems to work fine
My guess is this might be some kind of Mono bug, but I'm not sure how to debug this problem to find the cause, can anyone suggest the cause or something to research to try and fix these exceptions?
Here is an example exception:
2013-08-08 09:54:21.3991 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure ---> System.Net.WebException: Error getting response stream (ReadDone1): ReceiveFailure ---> System.IO.IOException: EndRead failure ---> System.Net.Sockets.SocketException: Connection reset by peer
at System.Net.Sockets.Socket.EndReceive (IAsyncResult result) [0x00000] in <filename unknown>:0
at System.Net.Sockets.NetworkStream.EndRead (IAsyncResult ar) [0x00000] in <filename unknown>:0
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.EndRead (IAsyncResult ar) [0x00000] in <filename unknown>:0
at Mono.Security.Protocol.Tls.SslStreamBase.InternalReadCallback (IAsyncResult result) [0x00000] in <filename unknown>:0
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult) [0x00000] in <filename unknown>:0
at System.Net.HttpWebRequest.GetResponse () [0x00000] in <filename unknown>:0
at Amazon.Runtime.AmazonWebServiceClient.getResponseCallback (IAsyncResult result) [0x00000] in <filename unknown>:0
--- End of inner exception stack trace ---
at Amazon.Runtime.AmazonWebServiceClient.handleHttpWebErrorResponse (Amazon.Runtime.Internal.AsyncResult asyncResult, System.Net.WebException we) [0x00000] in <filename unknown>:0
at Amazon.Runtime.AmazonWebServiceClient.getResponseCallback (IAsyncResult result) [0x00000] in <filename unknown>:0
And showing how often I get this error:
2013-08-07 20:31:04.4475 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 20:34:14.0017 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 20:45:06.9636 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 20:53:21.4654 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 20:56:16.8788 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 21:14:20.7060 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 21:21:17.3771 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 21:39:09.1383 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 21:57:11.6650 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 22:07:08.2615 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 22:44:16.6803 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 22:53:08.8771 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 22:53:09.1383 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-08 09:37:28.9755 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-08 09:42:28.3976 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-08 09:54:21.3991 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
This is the code that is trying to commit, the error is getting thrown by the ddb.BatchWriteItem call
private void BatchPutWithRetry(AmazonDynamoDBClient ddb, string table, IEnumerable<Dictionary<string, AttributeValue>> items)
{
var request = new BatchWriteItemRequest
{
RequestItems = new Dictionary<string, List<WriteRequest>>{
{table, items.Select(item => new WriteRequest{ PutRequest = new PutRequest{ Item = item }}).ToList()},
},
};
Log.Trace("BatchWriteItem {0}", request.RequestItems.First().Value.Count);
var totalSends = 0;
var unsentItems = request.RequestItems;
while (true)
{
try
{
totalSends++;
request.RequestItems = unsentItems;
var response = ddb.BatchWriteItem(request);
unsentItems = response.BatchWriteItemResult.UnprocessedItems;
}
catch (ProvisionedThroughputExceededException)
{
Log.Warn("ProvisionThroughputExceeded");
}
catch (Amazon.Runtime.AmazonServiceException ex)
{
if (totalSends > 1000)
throw;
Log.WarnException("Got service error", ex);
}
if (unsentItems.Count == 0)
return;
var delay = TimeSpan.FromMilliseconds(Random.Next(500, 3000));
Log.Trace("Unprocessed {0}, waiting {1:0.000} seconds",
unsentItems.First().Value.Count,
delay.TotalSeconds);
Thread.Sleep(delay);
}
}
I am running a thrift service on an EC2 instance AWS that is written in C# and running under Mono 3.2 (using the latest AWS .net SDK 1.5.28.2). The service hits dynamodb, and I am constantly getting low level networking errors when talking to dynamo. My current workaround is to just catch any AmazonServiceException and try again, which seems to work fine
My guess is this might be some kind of Mono bug, but I'm not sure how to debug this problem to find the cause, can anyone suggest the cause or something to research to try and fix these exceptions?
Here is an example exception:
2013-08-08 09:54:21.3991 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure ---> System.Net.WebException: Error getting response stream (ReadDone1): ReceiveFailure ---> System.IO.IOException: EndRead failure ---> System.Net.Sockets.SocketException: Connection reset by peer
at System.Net.Sockets.Socket.EndReceive (IAsyncResult result) [0x00000] in <filename unknown>:0
at System.Net.Sockets.NetworkStream.EndRead (IAsyncResult ar) [0x00000] in <filename unknown>:0
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.EndRead (IAsyncResult ar) [0x00000] in <filename unknown>:0
at Mono.Security.Protocol.Tls.SslStreamBase.InternalReadCallback (IAsyncResult result) [0x00000] in <filename unknown>:0
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult) [0x00000] in <filename unknown>:0
at System.Net.HttpWebRequest.GetResponse () [0x00000] in <filename unknown>:0
at Amazon.Runtime.AmazonWebServiceClient.getResponseCallback (IAsyncResult result) [0x00000] in <filename unknown>:0
--- End of inner exception stack trace ---
at Amazon.Runtime.AmazonWebServiceClient.handleHttpWebErrorResponse (Amazon.Runtime.Internal.AsyncResult asyncResult, System.Net.WebException we) [0x00000] in <filename unknown>:0
at Amazon.Runtime.AmazonWebServiceClient.getResponseCallback (IAsyncResult result) [0x00000] in <filename unknown>:0
And showing how often I get this error:
2013-08-07 20:31:04.4475 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 20:34:14.0017 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 20:45:06.9636 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 20:53:21.4654 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 20:56:16.8788 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 21:14:20.7060 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 21:21:17.3771 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 21:39:09.1383 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 21:57:11.6650 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 22:07:08.2615 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 22:44:16.6803 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 22:53:08.8771 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 22:53:09.1383 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-08 09:37:28.9755 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-08 09:42:28.3976 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-08 09:54:21.3991 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
This is the code that is trying to commit, the error is getting thrown by the ddb.BatchWriteItem call
private void BatchPutWithRetry(AmazonDynamoDBClient ddb, string table, IEnumerable<Dictionary<string, AttributeValue>> items)
{
var request = new BatchWriteItemRequest
{
RequestItems = new Dictionary<string, List<WriteRequest>>{
{table, items.Select(item => new WriteRequest{ PutRequest = new PutRequest{ Item = item }}).ToList()},
},
};
Log.Trace("BatchWriteItem {0}", request.RequestItems.First().Value.Count);
var totalSends = 0;
var unsentItems = request.RequestItems;
while (true)
{
try
{
totalSends++;
request.RequestItems = unsentItems;
var response = ddb.BatchWriteItem(request);
unsentItems = response.BatchWriteItemResult.UnprocessedItems;
}
catch (ProvisionedThroughputExceededException)
{
Log.Warn("ProvisionThroughputExceeded");
}
catch (Amazon.Runtime.AmazonServiceException ex)
{
if (totalSends > 1000)
throw;
Log.WarnException("Got service error", ex);
}
if (unsentItems.Count == 0)
return;
var delay = TimeSpan.FromMilliseconds(Random.Next(500, 3000));
Log.Trace("Unprocessed {0}, waiting {1:0.000} seconds",
unsentItems.First().Value.Count,
delay.TotalSeconds);
Thread.Sleep(delay);
}
}
0 commentaires:
Enregistrer un commentaire