Quantcast
Channel: Brian Pedersen's Sitecore and .NET Blog
Viewing all 285 articles
Browse latest View live

ASP.Net Core API – “‘s’ is an invalid start of a value. Path: $ | LineNumber: 0 | BytePositionInLine: 0.”

$
0
0

If you create an ASP.Net Core API Controller, and you wish to create an endpoint that can accept any string, you would expect this to be correct:

using Microsoft.ApplicationInsights;
using Microsoft.AspNetCore.Cors;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Configuration;
using Serilog;

namespace My.Api.Controllers
{
  [ApiController]
  [Route("/api/[controller]")]
  [Produces("application/json")]
  public class TestController : ControllerBase
  {
    [HttpPost()]
    [ProducesResponseType(200)]
    [ProducesResponseType(404)]
    public string PostTest([FromBody] string text)
    {
      Log.Information(text);
      return text;
    }
  }
}

But this endpoint will only accept strings in quotes, so posting string will not work, and you get the “‘s’ is an invalid start of a value. Path: $ | LineNumber: 0 | BytePositionInLine: 0.” error. But if you post “string”, the function works.

This is because the method accepts a JSON string, and not a raw string.

There is no direct way of allowing any string to be posted to your endpoint, as the endpoints always expects a type-strong value.

So what can you do?

OPTION 1: CHANGE THE INPUT TO OBJECT

If you know that the input is a JSON object, you can post an object instead:

[HttpPost()]
[ProducesResponseType(200)]
[ProducesResponseType(404)]
public string PostTest([FromBody] object text)
{
  Log.Information(text.ToString());
  return text.ToString();
}

This will accept any JSON object (for example { “a”: “b” }), but not a clean string.

OPTION 2: READ FROM THE REQUEST BODY DIRECTLY:

To accept any raw string, you need to indirectly read the request body:

[HttpPost()]
[ProducesResponseType(201)]
[ProducesResponseType(400)]
public async Task<IActionResult> PostTest()
{
  using (StreamReader reader = new StreamReader(Request.Body, Encoding.UTF8))
  {
    string message = await reader.ReadToEndAsync();
    return message;
  }
}

The downside of streaming the request body to a string is that you cannot test the endpoint using Swagger (or OpenApi as it is called today), as there is nowhere to write the request body. To test it you need a tool such as Postman or CURL.

MORE TO READ:

 


Sitecore use global and environment variables in config files

$
0
0

The Sitecore config files are a complex machine and it requires a healthy mind to work with it. Fortunately Sitecore have implemented a few tricks to ease our pain. This article focuses on 2 parts: global variable replacement and environment variable replacement.

GLOBAL VARIABLES:

Global variables are defined once and can then be reused in all your config settings.

Maybe you have seen these variables:

<sitecore>
  <sc.variable name="dataFolder" value="/App_Data" />
  <sc.variable name="mediaFolder" value="/upload" />
  <sc.variable name="tempFolder" value="/temp" />
</sitecore>

The sc.variable element defines the variable. The variables can then be used where you need them by writing $(variable_name):

<file value="$(dataFolder)/logs/myfile.log"/>
<setting name="IndexFolder" value="$(dataFolder)/indexes" />

The above variables are just the built in ones. You can create your own variables and use them too.

ENVIRONMENT VARIABLES:

Since Sitecore 8.2 we have been blessed with environment variable replacements too.

Let’s say you have an environment variable on your machine:

Global Environment Variables

Global Environment Variables

You can add the environment variable to the config by writing $(env:variable_name):

<setting name="Environment" value="$(env:ASPNETCORE_ENVIRONMENT)" />

This is especially nifty as you can deploy the same config file to different environments and control your application per server.

MORE TO READ:

 

.NET Core Catch Model Binding Exceptions

$
0
0

In .NET Core, if you create an API project, and you have an controller receiving an object you defined yourself, the controller will not be called if you use the wrong model, or if the model is not in a correct JSON/XML format.

This means that you cannot catch the exception in your own try/catch block, which is annoying if you would like to log the exception in your own log file, or to your own Application Insights instance.

To catch model binding exceptions, you need to configure the ApiBehaviorOptions in your ConfigureServices(IServiceCollection services) method.

STEP 1: CALL services.Configure<ApiBehaviorOptions>:

I have made a private method in my Startup.cs file:

private void ConfigureModelBindingExceptionHandling(IServiceCollection services)
{
  services.Configure&lt;ApiBehaviorOptions&gt;(options =>
  {
    options.InvalidModelStateResponseFactory = actionContext =>
    {
      ValidationProblemDetails error = actionContext.ModelState
          .Where(e => e.Value.Errors.Count > 0)
          .Select(e => new ValidationProblemDetails(actionContext.ModelState)).FirstOrDefault();

      // Here you can add logging to you log file or to your Application Insights.
	  // For example, using Serilog:
	  // Log.Error($"{{@RequestPath}} received invalid message format: {{@Exception}}", 
	  //   actionContext.HttpContext.Request.Path.Value, 
	  //   error.Errors.Values);
      return new BadRequestObjectResult(error);
    };
  });
}

STEP 2: CALL THE METHOD AFTER LOG AND/OR APPLICATION INSIGHTS HAVE BEEN INSTANTIATED:

Sequence is important. In the public void ConfigureServices(IServiceCollection services) method, after you have instantiated logging, Application Insights, or whatever you use. Then call the new method:

public void ConfigureServices(IServiceCollection services)
{
  // Add logging, Application Insights, Controllers, ...
  ...
  ...
  // Add exception logging on model binding exceptions
  ConfigureModelBindingExceptionHandling(services, telemetryClient);
  ...
  ...
}

MORE TO READ:

Sitecore and Application Insights – How to remove 90% of all log lines without sacrificing traceability

$
0
0

In this article I will explain how you can remove up to 90% of all log lines from the Application Insights log, but still keep every log line in the file log. All of this without loosing any important information.

Sitecore uses a Log4Net Appender to inject log lines into Application Insights. If you take a look at the App_Config/Sitecore/Azure/Sitecore.Cloud.ApplicationInsights.config file that is deployed to your CM/CD servers, you will find the appender at the top, and it could look like this:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:env="http://www.sitecore.net/xmlconfig/env/" xmlns:role="http://www.sitecore.net/xmlconfig/role/"> 
<sitecore> 
  <log4net> 
    <root> 
      <appender-ref ref="ApplicationInsightsAppender" patch:after="appender-ref[@ref='LogFileAppender']" />
    </root> 
    <appender name="ApplicationInsightsAppender" type="Sitecore.Cloud.ApplicationInsights.Logging.LevelTraceAppender, Sitecore.Cloud.ApplicationInsights" patch:after="appender[@name='LogFileAppender']"> 
      <threshold value="INFO" /> 
        <layout type="log4net.Layout.PatternLayout"> 
          <conversionPattern value="%4t %d{ABSOLUTE} %-5p %m%n" />
        </layout> 
    </appender> 
  </log4net> 
</sitecore> 
</configuration>

Sitecore does log a lot of information that probably should have been marked as debug information. This is fine when logging to a log file. But with Application Insights there are issues with logging too much information:

  • You have to pay for what you log. Although Azure is priced reasonable, it is not free. Any excessive logging can make your budget explode.
  • You might run into  sampling issues (“Data received from your application is being sampled to reduce the volume of telemetry data retained; only sampled documents will be returned. The sampling may be applied by the Application Insights SDK or on ingestion by Application Insights.“). This can be disabled, but sampling usually are there for a reason, as Application Insights cost money the more lines you log.

When I analyze my log files, there are several lines that have never helped me:

ManagedPoolThread #6 00:02:46 INFO Job started: Job started: Index_Update_IndexName=sitecore_Master_index
ManagedPoolThread #9 00:04:48 INFO Scheduling.DatabaseAgent started. Database: master
ManagedPoolThread #13 00:04:48 INFO Cache created: 'ExperienceAnalytics.DimensionItems' (max size: 2MB, running total: 9090MB)

The “Job started” alone can take up to 90% of the log lines in a solution!

To remove these log lines from the Application Insights log, we can use the log4net.Filter.StringMatchFilter. This filter can add or remove lines containing a string.

Filters are added to the appender attribute in the config file like this:

<log4net>
  <root>
    <appender-ref ref="ApplicationInsightsAppender" patch:after="appender-ref[@ref='LogFileAppender']" />
  </root>
  <appender name="ApplicationInsightsAppender" type="Sitecore.Cloud.ApplicationInsights.Logging.LevelTraceAppender, Sitecore.Cloud.ApplicationInsights" patch:after="appender[@name='LogFileAppender']">
    <threshold value="INFO" />
    <layout type="log4net.Layout.PatternLayout">
      <conversionPattern value="%4t %d{ABSOLUTE} %-5p %m%n" />
    </layout>
    <filter type="log4net.Filter.StringMatchFilter">
      <stringToMatch value="Job started: Index_Update_IndexName=" />
      <acceptOnMatch value="false" />
    </filter>
    ...
    ...
  </appender>
</log4net>

The “stringToMatch” is the string that Log4Net looks for, and the acceptOnMatch=false ensure that the log line is not appended to the log.

I have found the following candidates for removal from the Application Insights log:

  • Job started: Index_Update_IndexName=
  • Job ended: Index_Update_IndexName=
  • Job started: UpdateIndexDocuments
  • Job ended: UpdateIndexDocuments
  • Job started: Sitecore.
  • Job ended: Sitecore.
  • Request is redirected to no layout page
  • HttpModule is being initialized
  • File is being deleted by cleanup task
  • Cache created:
  • Health.

There are probably more depending on your Sitecore installation. You should use the old but excellent Sitecore Log Analyzer to analyse the file logs and find the repetitive log lines that you never use, and filter them from the Application Insights log. Remember that what does not help me, might help you. But removing noise really does give a better overview, and your client’s wallet will be happy too :).

MORE TO READ:

 

.NET Core Api – Catch exceptions using a middleware

$
0
0

In .NET Core, we are starting to get used to the fact that nothing comes for free, and in the world of ultimate freedom of choice, every feature needs to be implemented by us.

This includes error handling. In a previous post, .NET Core Catch Model Binding Exceptions, I explained how you can use ApiBehaviorOptions to catch exceptions thrown when someone POST the wrong JSON format to your endpoint.

It should be noted, that you can still catch exceptions in your endpoint using try…catch:

namespace MyCode.Controllers
{
  [ApiController]
  [Route("/api")]
  [Produces("application/json")]
  public class MyController : ControllerBase
  {
    [HttpPost()]
    public IActionResult LogDoNavigate([FromBody] MyModel model)
    {
      try
      {
	    ...
            ...
      }
      catch (Exception ex)
      {
        return base.BadRequest(new ProblemDetails() { Title = ex.Message, Detail = ex.ToString() });
      }
    }
  }
}

But what if the exception is not in this part of the Controller? What if the exception is in a System.Text.Serializion.JsonConverter that runs before your endpoint is called? The code will still throw an exception of course, but there is no immediate default handler, and the exception will not end up in your log file, or your Application Insights instance.

To handle such a scenario, you need to implement a Middleware. Middlewares are a piece of code that handles request and responses. You can use middlewares for many situations, but the most obvious are as exception handlers.

STEP 1: CREATE A CUSTOM MIDDLEWARE

This middleware will catch errors and return the exception as a JSON result object. You need to add your own logging or Application Insights code to write to log.

using Microsoft.AspNetCore.Diagnostics;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;
using System;
using System.ComponentModel.DataAnnotations;
using System.Net;
using System.Security.Authentication;
using System.Threading.Tasks;
using Newtonsoft.Json;

namespace MyCode
{
  public class JsonExceptionMiddleware
  {
    public JsonExceptionMiddleware()
    {
    }

    public async Task Invoke(HttpContext context)
    {
      var contextFeature = context.Features.Get<IExceptionHandlerFeature>();
      if (contextFeature != null && contextFeature.Error != null)
      {
        // ...
        // Add lines to your log file, or your 
        // Application insights instance here
        // ...
        context.Response.StatusCode = (int)GetErrorCode(contextFeature.Error);
        context.Response.ContentType = "application/json";

        await context.Response.WriteAsync(JsonConvert.SerializeObject(new ProblemDetails()
        {
          Status = context.Response.StatusCode,
          Title = contextFeature.Error.Message
        }));
      }
    }

    private static HttpStatusCode GetErrorCode(Exception e)
    {
      switch (e)
      {
        case ValidationException _:
          return HttpStatusCode.BadRequest;
        case FormatException _:
          return HttpStatusCode.BadRequest;
        case AuthenticationException _:
          return HttpStatusCode.Forbidden;
        case NotImplementedException _:
          return HttpStatusCode.NotImplemented;
        default:
          return HttpStatusCode.InternalServerError;
      }
    }
  }
}

STEP 2: ADD THE MIDDLEWARE TO YOUR IApplicationBuilder

Add the middleware to the Configure method in the Startup.cs file. Call the UseExceptionHandler method to use the middleware:

public void Configure(IApplicationBuilder app)
{
  ...
  ...
  app.UseExceptionHandler(new ExceptionHandlerOptions { 
    ExceptionHandler = new JsonExceptionMiddleware().Invoke 
    }
  );
  ...
  ...
}

You would expect the middleware to catch any exception. It does catch exceptions in your Controller code, so don’t need a try-catch handler in every Controller. But it does not catch model binding exceptions, so you still need to catch those.

MORE TO READ:

Sitecore use item:deleting event to avoid deletion of items

$
0
0

The Sitecore extendable model allows you to build business rules into your Sitecore solution. This is an example where I disallow deletion of certain items when a certain business rule is true.

This pattern could be implemented using a field or template validator, or by using the Sitecore workflow,  but it would require you to remember to add the validator to the correct template, and also requires that no one tampers with your template settings.

Sitecore have an item:deleting event that is fired before an item is deleted. The arguments include a Result.Cancel property that can be set to false, thus stopping the deletion of your item. So the implementation looks like this:

using System;
using Sitecore.Data.Items;
using Sitecore.Events;
using Sitecore.Web.UI.Sheer;

namespace MyProject
{
  public class OnItemDeletingValidator
  {
    public void OnItemDeleting(object sender, EventArgs args)
    {
      // get the item that is being moved
      Item item = Event.ExtractParameter(args, 0) as Item;
      
      if (item == null)
        return;

      if (item.Database.Name != "master")
        return;

      // Only run the rule on a certain template.
      // Replace the "MyTemplate" with the name of your own
      // template
      if (item.TemplateName != "MyTemplate")
        return;

      // Check to see if the business rule is true or false
      // If true, do nothing.
      // Implemement your own business rule here
      if (BusinessRule(item))
        return;
      
      // Alert the user and cancel the delete
      SheerResponse.Alert("You cannot delete this item because the business rule is false");
      ((SitecoreEventArgs)args).Result.Cancel = true;
    }
  }
}

The code above is just a pseudo code example. You need to implement your own business rule, and determine your own template.

Also remember to assign the code to the event using configuration:

<event name="item:deleting">
  <handler type="MyProject.OnItemDeletingValidator, MyProject" method="OnItemDeleting"/>
</event>

MORE TO READ:

Sitecore Create/Read/Update/Delete/Copy/Move/Rename item – the beginners guide

$
0
0

New Sitecorians often ask the most trivial questions, and I am happy to answer them. This question popped up lately: How do you perform basic CRUD operations on your Sitecore items? Well, it’s easy:

READ ITEM (GET ITEM):

// Get item from content database:
Item item = Sitecore.Context.ContentDatabase.GetItem("/sitecore/content/home");
// Get item from ID:
Item item = Sitecore.Context.ContentDatabase.GetItem(new Sitecore.Data.ID("{9464f2c9-8490-40e9-a95b-17f8a5128da6}");
// Get item from named database
Item item = Sitecore.Configuration.Factory.GetDatabase("master").GetItem("/sitecore/content/home");

CREATE ITEM:

// You need to have a root item to create item under:
Item item = Sitecore.Context.ContentDatabase.GetItem("/sitecore/content/home");
// You also need a template to create the item from:
TemplateID template = new TemplateID(new ID("{434b38a2-b929-4a89-bbc8-a6b66281e014}"));
// Then you can create a new item:
Item newItem = item.Add("new item", template);
// If you wish to create an item based on a branch:
BranchId branch = new BranchId(new ID("{4f254169-7666-4c2e-8021-a05026d5a2e2}"));
Item newItem = item.Add("new item", branch);

UPDATE ITEM:

// You need to have an item to update:
// Remember to always update items in the MASTER database,
// NOT the WEB Database:
Item item = Factory.GetDatabase("master").GetItem("/sitecore/content/home");
// You then set the item in editing mode
item.Editing.BeginEdit();
try
{ 
  // Change the contents of the fields to update
  item.Fields["field"].Value = "new value";
  // End edit writes the updates to the database:
  item.Editing.EndEdit();
}
catch (Exception ex)
{
  // in case of an exception, you do not really
  // need to cancel editing, but it is good 
  // manners and it indicates that you know
  // what the code is doing
  item.Editing.CancelEdit();
}

DELETE ITEM:

// You need to have an item to delete:
// Remember to always update items in the MASTER database,
// NOT the WEB Database:
Item item = Factory.GetDatabase("master").GetItem("/sitecore/content/home");
// Remember that deleting one item also delete the children.
// The item.Recycle() moves the item to the recycle bin, whilst the 
// item.Delete() permanently deletes the item:
item.Recycle();

COPY ITEM:

// You need to have a source destination:
Item destinationItem = Factory.GetDatabase("master").GetItem("/sitecore/content/home");
// You also need an item to copy:
Item sourceItem = Factory.GetDatabase("master").GetItem("/sitecore/content/sourceitem");
// Then you can copy:
sourceItem.CopyTo(destinationItem, "new name");

MOVE ITEM:

// You need to have a source destination:
Item destinationItem = Factory.GetDatabase("master").GetItem("/sitecore/content/home");
// You also need an item to move:
Item sourceItem = Factory.GetDatabase("master").GetItem("/sitecore/content/sourceitem");
// Then you can copy:
sourceItem.MoveTo(destinationItem);

RENAME ITEM:

// You need to have an item to rename:
Item item = Factory.GetDatabase("master").GetItem("/sitecore/content/home");
item.Editing.BeginEdit();
item.Name = "new name";
item.Editing.EndEdit();

MORE TO READ:

Command line parameters in .NET Core Console Applications

$
0
0

When implementing .NET Core console applications, you can extract the command line parameters from the void Main(string[] args) method, but it can be a little tedious to do so. To help with the parsing, Microsoft used to maintain the Microsoft.Extensions.CommandLineUtils, but this library have not been in development since 2017, so this is not a good option.

Thankfully, Nate McMaster forked the Microsoft project and created the CommandLineUtils, so developers like you and I don’t have to concern us with parsing of parameters, creating help pages, documentation and so on.

First you need to install the NuGet package:

McMaster.Extensions.CommandLineUtils

This is a base template of how your Program.cs main entry point will look like when using CommandLineUtils:

using McMaster.Extensions.CommandLineUtils;

namespace MyApplication
{
  [Command(Name = "The application name", Description = "The application description")]
  [HelpOption("-?")]
  public class Program
  {
    static void Main(string[] args) => CommandLineApplication.Execute<Program>(args);

    [Argument(0, Description = "The first argument")]
    private string MyArgument { get; }

    [Option("-o|--option", Description = "An example parameter")]
    private bool MyOption { get; }
    
    private void OnExecute()
    {
      // Initialize all the things...
      // Configure logging, application insights, ...
      // Run your application
    }
  }
}

EXPLANATION: THE Main(string[] args) METHOD:

This function is now pointing to the “OnExecute()” function instead. Treat theOnExecute() method as you would treat the Main(string[] args) method and initialize the servicecollection, logging, dependencies as you would normally do it.

EXPLANATION: ATTRIBUTE [Argument(…..)] VS [Option(……)]

CommandLineUtils treat command line parameters differently.

The [Argument] attribute determines parameters without any prefix. They also have a fixed position in the list of parameters.

The the example above, the MyArgument MUST be the first parameter when the application is called. This is determined by the value zero (0) in the attribute:

[Argument(0, Description = "The first argument")]

The value of MyArgument is set only by the first parameter:

./myapplication.exe firstparameter

Here, the value of MyArgument will be “firstparameter“.

The [Option] attribute does not have a fixed position, but a prefix instead. The boolean value of MyOption will be FALSE if -o or -option is NOT set, but true if -o or -option is set:

./myapplication.exe -o

Here, the value of MyOption is TRUE.

Options can have string parameters as well:

[Option("-o2|--option2", Description = "Another Option")]
private string MyOption2 { get; } = "Hello World";

The value of MyOption2 is “Hello world” unless you set the value in a parameter:

./myapplication.exe -o2 AnotherValue

Here, the value of MyOption2 will be “AnotherValue“.

The library have tons of more options, from automatic help pages, prompts for yes/no/passwords, optional/required parameters and so on. Dig into the library and see for yourself. Happy coding.

MORE TO READ:


Azure App Configuration – Getting the connection string from appsettings.json

$
0
0

Azure App Configuration is a services that provides a central place to manage application settings, and it provides cool features like feature flags and auto refresh of settings when they are changed.

Azure App Configuration

Azure App Configuration

To access Azure App Configuration from your .NET Core application, you include the preview NuGet Package:

Microsoft.Extensions.Configuration.AzureAppConfiguration

PLEASE NOTE: As per 2020-02-14, Azure App Configuration is still in beta. You must check the “include prerelease” button in Visual Studio to see the NuGet package.

To connect to the Azure App Configuration you need a connection string.

Most of the guides and help pages expects you get the connection string from an environment variable, or that you create a web host application and can get the connection string from a web config file.

But if you would like to use Azure App Configuration together with a local appsettings.json file, there is no help available.

The challenge is that the connectionstring to the Azure App Configuration is not available before you read and build a IConfiguration reading from the local appsettings.json. So you need some way to split the calls to ConfigurationBuilder and then merge the 2 together.

THE SCENARIO:

Let’s assume that you have a local appsettings.json file and that file includes the connectionstring (and maybe even some more settings that are not to be tampered with by those with access to the Azure App Configurations):

{
  "Logging": {
    "FileLogPath": "e:\somewhere\log_.txt"
  },
  "ConnectionStrings": {
    "AzureAppConfiguration": "Endpoint=https://xxxxxxxx.azconfig.io;Id=xxxxxxxx;Secret=xxxxxxxx",
  }
}

THE SOLUTION:

You then need to merge these settings with the settings from Azure App Configuration.

var localConfig = new ConfigurationBuilder()
 .SetBasePath(Directory.GetCurrentDirectory())
 .AddJsonFile("appsettings.json", false, true)
 .Build();

var configuration = new ConfigurationBuilder()
 .AddConfiguration(localConfig)
 .AddAzureAppConfiguration(
    options =>
    {
      options.Connect(configuration.GetConnectionString("AzureAppConfiguration"));
    }
  )
 .Build();

The localConfig variable contains the settings from appsettings.json. This configuration is then merged with the Azure App Configuration into the configuration variable, which is then the final IConfiguration that will be used in the application.

MORE TO READ:

Application Insights not working in .NET Core Console Application – Remember to flush the TelemetryClient

$
0
0

My .NET Core console application did not write to my Application Insights instance. Well, it did, but only sometimes.

Writing to Application Insights is done asynchronously, usually every 30 seconds or 500 items. This means that you need to give your application some time to flush the cache before closing down. This can be done by configuring a ProcessExit method. This method is called when your Main() function exists.

This is a template where I initialize my TelemetryClient, and then flushes and waits 5 seconds for the cache to flush before the application is exited:

using System;
using System.Threading;
using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.Extensibility;
using Microsoft.Extensions.DependencyInjection;

namespace MyCode
{
  internal class Program
  {
    public static void Main(string[] args)
    {
      // Initialize Service Collection
      var serviceCollection = new ServiceCollection();
      ConfigureServices(serviceCollection);

      // Initialize Service Provider
      var serviceProvider = serviceCollection.BuildServiceProvider();

      // Handling finalizing when process is ended
      AppDomain.CurrentDomain.ProcessExit += (s, e) => FinalizeApplication(serviceProvider);

      // Run Application
      // ... do run ...
    }

    private static void FinalizeApplication(ServiceProvider serviceProvider)
    {
      // Give TelemetryClient 5 seconds to flush it's content to Application Insights
      serviceProvider.GetService<TelemetryClient>().Flush();
      Thread.Sleep(5000);
    }

    private static void ConfigureServices(IServiceCollection serviceCollection)
    {
      // Add Application Insights
      var telemetryConfiguration = TelemetryConfiguration.CreateDefault();
      telemetryConfiguration.InstrumentationKey = "add-your-own-key";
      var telemetryClient = new TelemetryClient(telemetryConfiguration);
      serviceCollection.AddSingleton(telemetryClient);
    }

  }
}

HOW IT WORKS:

My .NET Core 3.1 console application is started by the Main() function. It starts by initializing the service collection. The only thing I do is adding a TelemetryClient to the collection, but this is where you would all all of your services.

The AppDomain.CurrentDomain.ProcessExit is configured to call the “FinalizeApplication” method, and this method will now be called whenever the Main() function exits.

The FinalizeApplication will then flush the TelemetryClient, and waits 5 seconds for the flush to do it’s magic.

Without the 5 second wait, I risk that the Flush is not done flushing before the thread is discontinued, hence the wait. For some reason, 5 seconds seems to be the magic threshold for a flush to do it’s thing.

MORE TO READ:

C# Get expiry timestamp from JWT token

$
0
0

JWT tokens (or Json Web Tokens) are an open-standard the defines a way to transmit information between 2 parties in a secure manner. Identity Server 4 uses JWT as a security token.

These tokens have an expiry timestamp, and if you handle the tokens yourself, you need to read the token expiry and refresh the token if the token is expired.

Microsoft have made a brilliant library, System.IdentityModel.Tokens.Jwt to handle JWT tokens, but the package does also have a lot of dependencies that were incompatible with my application, so I chose to use JWT.Net instead, as this package does not have any dependencies at all.

THE ANATOMY OF A JWT TOKEN:

Json Web Token Anatomy

Json Web Token Anatomy

A JWT token consists of a header, a payload and a signature. It is in the payload that you find the expiry timestamp in the “exp” field. The timestamp is the stupid UNIX timestamp format, but fear not, .NET knows how to convert the timestamp to a real DateTime.

STEP 1: CREATE A PAYLOAD MODEL CLASS

JWT.Net is not as powerful as System.IdentityModel.Tokens.Jwt, so you need to create a model class of the payload section. The class, however, is very simple:

namespace MyCode
{
  public class JwtToken
  {
    public long exp { get; set; }
  }
}

STEP2: USE JWT.Net TO GET THE EXPIRY FROM THE TOKEN PAYLOAD

Final step is to take the JWT Token string and decode it to the JwtToken class, then convert the UNIX timestamp to a local time:

using System;
using JWT;
using JWT.Algorithms;
using JWT.Serializers;

namespace MyCode
{
  public class JWTService
  {
    private IJsonSerializer _serializer = new JsonNetSerializer();
    private IDateTimeProvider _provider = new UtcDateTimeProvider();
    private IBase64UrlEncoder _urlEncoder = new JwtBase64UrlEncoder();
    private IJwtAlgorithm _algorithm = new HMACSHA256Algorithm();

    public DateTime GetExpiryTimestamp(string accessToken)
    {
      try
      {
        IJwtValidator _validator = new JwtValidator(_serializer, _provider);
        IJwtDecoder decoder = new JwtDecoder(_serializer, _validator, _urlEncoder, _algorithm);
        var token = decoder.DecodeToObject<JwtToken>(accessToken);
        DateTimeOffset dateTimeOffset = DateTimeOffset.FromUnixTimeSeconds(token.exp);
        return dateTimeOffset.LocalDateTime;
      }
      catch (TokenExpiredException)
      {
        return DateTime.MinValue;
      }
      catch (SignatureVerificationException)
      {
        return DateTime.MinValue;
      }
      catch (Exception ex)
      {
        // ... remember to handle the generic exception ...
        return DateTime.MinValue;
      }
    }
  }
}

That’s it. You are now a security expert. Happy coding.

FUNNY FINAL NOTE:

The term “JWT Token” is a redundant acronym syndrome, or RAS-syndrome. It is the use of the last word of the acronym in conjunction with the abbreviated form. It’s like saying “PIN number” or “PDF format”. In reality, when saying “JWT Token”, you are really saying “json web token token” :).

MORE TO READ:

Sync Sitecore content to external database

$
0
0

The legal department of the client requested that we sync parts of the Sitecore tree to an external, system-versioned temporal table, so that we have a full audit trail of changes to these items including before-and-after values.

System-Versioned Table

Example of a System-Versioned Table. When looking into the History you can see a full audit trail of the Sitecore items

Fear not, this is what Sitecore does best. All we need to do is to create a SQL server database with temporal tables, create upsert (a combined insert and update statement) and delete Stored Procedures, and then hook into the Sitecore item:created, item:renamed, item:saved, item:moved and item:deleted events.

STEP 1: THE BASIC DATABASE PATTERN

First we need to create a table for the data. Maybe you like to create your tables in a different matter, and that is OK, but this is the minimum required fields needed to perform a data audit:

CREATE TABLE [dbo].[MyItem]
(
    [SitecoreItemID] UNIQUEIDENTIFIER NOT NULL PRIMARY KEY,
	...
	... here you define the fields from the Sitecore template
	...
    [Created] DATETIME NOT NULL DEFAULT GetDate(), 
    [Updated] DATETIME NULL, 
    [Deleted] DATETIME NULL, 
    [IsDeleted] BIT NOT NULL, 
    [SitecoreUserName] NVARCHAR(255) NOT NULL,
    [SysStartTime] DATETIME2 GENERATED ALWAYS AS ROW START NOT NULL,
    [SysEndTime] DATETIME2 GENERATED ALWAYS AS ROW END NOT NULL,
    PERIOD FOR SYSTEM_TIME (SysStartTime,SysEndTime)
    )
WITH (SYSTEM_VERSIONING = ON (HISTORY_TABLE = dbo.MyItem));
  • [SitecoreItemID]
    Is the key value is the Sitecore item id. Some like to have an auto-incrementing integer as key, and that’s also fine, as long as you at least have the Sitecore item ID in the database.
  • [Created], [Updated], [Deleted]
    Timestamps are used to determine when database operations are carried out
  • [IsDeleted]
    Instead of deleting items, we mark them as deleted. The [Deleted] timestamp will tell us when the item was deleted.
  • [SitecoreUsername]
    Is the name of the Sitecore editor creating/updating/deleting the item
  • [SysStartTime],[SysEndTime]
    Is used by SQL to handle the system-versioning.
  • Remember to add all the fields from your Sitecore template that needs to be synchronized.

Also note that in my example, I do not store the item hierarchy, item languages or item versions.

STEP 2: UPSERT AND DELETE STORED PROCEDURES:

The basic UPSERT Stored procedure is as follows:

  • Select from database with [SitecoreItemID]
  • If found: Update
    Update with values from the Sitecore template
    Set [Updated] to current datetime
    Set [SitecoreUserName] with the Sitecore username
  • If not found: Insert
    Insert values from the Sitecore template
    Set [SitecoreUserName] with the Sitecore username

The basic DELETE Stored procedure is as follows:

  • Set [IsDeleted] to TRUE
    Set [SitecoreUserName] with the Sitecore username
    Set [Deleted] with the current datetime.

STEP 3: HOOK INTO SITECORE 

Now for the fun part, the Sitecore code.

We need to hook into the following Sitecore events:

  • item:created: Call the UPSERT stored procedure
  • item:renamed:Call the UPSERT stored procedure
  • item:saved: Call the UPSERT stored procedure
  • item:moved: Call the UPSERT stored procedure
  • item:deleted: Call the DELETE stored procedure

My code will only sync one language and always the latest version. If you need to sync more languages or versions, remember to add those as key values to your database.

First the code. This is a pseudo-code example, outlining what you need to implement.

using System;
using Sitecore.Data.Items;
using Sitecore.Events;

namespace MyCode
{
  public class SyncItems
  {
    public void OnItemChanged(object sender, EventArgs args)
    {
      Item changedItem = Event.ExtractParameter(args, 0) as Item;
      if (changedItem == null)
        return;
      // We only synchronize one language and only the latest version:
      if (changedItem.Language != Language.Parse("en"))
        return false;
      if (!changedItem.Versions.IsLatestVersion())
        return false;
        
      // If changed item is of the template we wish to synchronize:
      // Call the upsert stored procedure
    }

    public void OnItemMoved(object sender, EventArgs args)
    {
      Item movedItem = Event.ExtractParameter(args, 0) as Item;
      if (movedItem == null)
        return;
      // We only synchronize one language and only the latest version:
      if (movedItem.Language != Language.Parse("en"))
        return false;
      if (!movedItem.Versions.IsLatestVersion())
        return false;

      // If moved item is of the template we wish to synchronize:
      // Call the upsert stored procedure
    }
    
    public void OnItemDeleted(object sender, EventArgs args)
    {
      Item deletedItem = Event.ExtractParameter(args, 0) as Item;
      if (deletedItem == null)
        return;
      // We only synchronize one language and only the latest version:
      if (deletedItem.Language != Language.Parse("en"))
        return false;
      if (!deletedItem.Versions.IsLatestVersion())
        return false;

      // If deleted item is of the template we wish to synchronize:
      // Call the delete stored procedure
    }
}   

Then the Sitecore configuration:

<?xml version="1.0" encoding="utf-8"?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:role="http://www.sitecore.net/xmlconfig/role/" xmlns:env="http://www.sitecore.net/xmlconfig/env/">
  <sitecore>
    <events>
      <event name="item:created">
        <handler type="MyCode.SyncItems, MyCode" method="OnItemChanged"/>
      </event>
      <event name="item:renamed">
        <handler type="MyCode.SyncItems, MyCode" method="OnItemChanged"/>
      </event>
      <event name="item:saved">
        <handler type="MyCode.SyncItems, MyCode" method="OnItemChanged"/>
      </event>
      <event name="item:moved">
        <handler type="MyCode.SyncItems, MyCode" method="OnItemMoved"/>
      </event>
      <event name="item:deleted">
        <handler type="MyCode.SyncItems, MyCode" method="OnItemDeleted"/>
      </event>
    <events>
  </sitecore>
</configuration>

THINGS TO CONSIDER:

Should you use a synchronous or asynchronous update method?

If you call the stored procedures synchronously, all code will wait for and answer from the  SQL server. If you a asynchronous approach, you don’t catch exceptions from the database. It is up to you how critical errors are. There are coding patterns that will give you a more foul proof approach such as database retry mechanisms, or implementing a queue in between Sitecore and the database. You decide what is best.

What happens if you move an item with children?

If you are synchronizing items with children, and you stored the hierarchy of items in the database, you will need to make a cascading update on those children. Sitecore will not throw individual events for the children, only the root item that is moved.

What happens if users restore items from the recycle bin?

This is a situation that is currently undetermined, as there is no event attached to the recycle bin restore.

Are there any benefits? Why don’t just use Sitecore versioning and workflows?

Workflows are designed to enforce business rules upon editors, and implies a very stringent way of working. Workflows discourages casual editing and is often abandoned for that very reason. You should decide if a simple workflow would work for you.

MORE TO READ:

System.DllNotFoundException: Unable to load DLL ‘sni.dll’ or one of its dependencies: The specified module could n ot be found. (0x8007007E)

$
0
0

This message happens when deploying my .NET Core 3.1 application to production (when compiling the code to an .exe file), but not when running the application locally.

It turns out, that Dapper is missing a reference to System.Data.SqlClient. Adding the System.Data.SqlClient NuGet package to the project solves the issue:

System.Data.SqlClient

System.Data.SqlClient

The full error message is:

Unhandled exception. Unhandled exception. System.TypeInitializationException: The type initializer for ‘System.Data.SqlClient.TdsParser’ threw an exception.
—> System.TypeInitializationException: The type initializer for ‘System.Data.SqlClient.SNILoadHandle’ threw an exception.
—> System.DllNotFoundException: Unable to load DLL ‘sni.dll’ or one of its dependencies: The specified module could not be found. (0x8007007E)
at System.Data.SqlClient.SNINativeMethodWrapper.SNIInitialize(IntPtr pmo)
at System.Data.SqlClient.SNINativeMethodWrapper.SNIInitialize()
at System.Data.SqlClient.SNILoadHandle..ctor()
at System.Data.SqlClient.SNILoadHandle..cctor()
— End of inner exception stack trace —
at System.Data.SqlClient.TdsParser..cctor()
— End of inner exception stack trace —
at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObject
sTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& co
nnection)
at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry
, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`
1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
at System.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions) at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
at System.Data.SqlClient.SqlConnection.Open()
at Dapper.SqlMapper.QueryImpl[T](IDbConnection cnn, CommandDefinition command, Type effectiveType)+MoveNext() in C:\projects\dapper\Dapper\SqlMapper.cs:line 1079
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
at Dapper.SqlMapper.Query[T](IDbConnection cnn, String sql, Object param, IDbTransaction transaction, Boolean buffered, Nullable`1 commandTimeout, Nullable`1 commandType) in C:\projects\dapper\Dapper\SqlMapper.cs:line 721

MORE TO READ:

.NET Core Worker Services with Application Insights and Serilog

$
0
0

The .NET Core Worker service is yet another tool in the .NET toolbox. They are perfect for background processing like reading from a queue or making health checks. They are cross-platform (of course) and they can run on docker containers, and even run on a locally hosted machine as a Windows service.

It took me quite some time to implement Application Insights and file logging (Serilog) in my worker service, so I thought I might write it down, in case I need it another time.

THE NUGET PACKAGES

First caveat was when I realized that you need a special Microsoft.ApplicationInsights.WorkerService NuGet package before Application Insights will work. Also, if you wish to run the worker service as a Windows service you need the Microsoft.Extensions.Hosting.WindowsServices package.

The packages I needed was:

Microsoft.ApplicationInsights.AspNetCore
Microsoft.ApplicationInsights.WorkerService
Microsoft.Extensions.Hosting
Microsoft.Extensions.Hosting.WindowsServices
Serilog
Serilog.AspNetCore
Serilog.Extensions.Hosting
Serilog.Extensions.Logging
Serilog.Settings.Configuration
Serilog.Sinks.Console
System.Configuration.ConfigurationManager

THE APPSETTINGS.JSON CONFIGURATION

The configuration is pretty straight forward. Only caveat is that Serilog is not configured under “Logging” but under it’s own namespace:

{
  "ApplicationInsights": {
    "InstrumentationKey": "the-appinsights-guid"
  },
  "Serilog": {
    "WriteTo": [
      {
        "Name": "File",
        "Args": {
          "path": "log_.txt",
          "rollingInterval": "Day"
        }
      }
    ]
  },
  "Logging": {
    "ApplicationInsights": {
      "LogLevel": {
        "Default": "Information",
        "Microsoft": "Warning"
      }
    },
    "LogLevel": {
      "Default": "Information",
      "Microsoft": "Information",
      "Microsoft.Hosting.Lifetime": "Warning"
    }
  }
}

THE PROGRAM.CS

The examples online was too complex for me, so I simplified it and came up with this:

namespace MyWorkerService
{
  public class Program
  {
    public static void Main(string[] args)
    {
      // This will run the worker service
      CreateHostBuilder(args).Build().Run();
    }

    public static IHostBuilder CreateHostBuilder(string[] args)
    {
      var host = Host.CreateDefaultBuilder(args);
      // Add this line to be able to run as a windows service
      // https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/windows-service?view=aspnetcore-3.1&tabs=visual-studio
      host.UseWindowsService();
      // This will add the configuration to the application.
      // It will allow you to inject the configuration as 
      // IConfiguration configuration in your constructors 
      host.ConfigureAppConfiguration(
            (hostContext, config) =>
            {
              config.SetBasePath(Directory.GetCurrentDirectory());
              config.AddJsonFile("appsettings.json", false, true);
              config.AddCommandLine(args);
            }
      );
      // This will configure logging. It reads the log settings from 
      // the appsettings.json configuration file, and adds serilog,
      // allowing the application to write logs to file
      host.ConfigureLogging(
            loggingBuilder =>
            {
              var configuration = new ConfigurationBuilder()
                 .AddJsonFile("appsettings.json")
                 .Build();
              var logger = new LoggerConfiguration()
                  .ReadFrom.Configuration(configuration)
                  .CreateLogger();
              loggingBuilder.AddSerilog(logger, dispose: true);
            }
      );
      // The AddHostedServer adds the worker service to the application.
      // The AddApplicationInsightsTelemetryWorkerService is very important. Without this,
      // the Application Insights logging will not work.
      host.ConfigureServices((hostContext, services) =>
      {
        services.AddHostedService<Worker>();
        services.AddApplicationInsightsTelemetryWorkerService();
      });

      return host;
    }
  }
}

The worker itself is the “Worker” class and it looks like this:

using System.Threading;
using System.Threading.Tasks;
using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.DataContracts;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Configuration;

namespace MyWorkerService
{
  public class Worker : BackgroundService
  {
    private readonly ILogger<Worker> _logger;
    private readonly TelemetryClient _telemetryClient;

    public Worker(ILogger<Worker> logger,
      TelemetryClient telemetryClient,
      IConfiguration configuration)
    {
      _logger = logger;
      _telemetryClient = telemetryClient;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
      while (!stoppingToken.IsCancellationRequested)
      {
        using (_telemetryClient.StartOperation<RequestTelemetry>("Execute Async"))
        {
          _logger.LogInformation("ExecuteAsync Started");
          DoWork();
        }
      }
      await Task.Delay(1000, stoppingToken);
    }
  }
}

The DoWork() is where you would do actual work.

MORE TO READ:

HttpClient retry mechanism with .NET Core, Polly and IHttpClientFactory

$
0
0

A lot of HttpClient errors are temporary and is caused by server overload, temporary nerwork timeouts and generic gliches in the Matrix. These scenarios can be dealt with using a retry pattern. In .NET Core, the most common retry library is the Polly library:

Polly is a .NET resilience and transient-fault-handling library that allows developers to express policies such as Retry, Circuit Breaker, Timeout, Bulkhead Isolation, and Fallback in a fluent and thread-safe manner. From version 6.0.1, Polly targets .NET Standard 1.1 and 2.0+.
http://www.thepollyproject.org/

Polly makes it relatively easy to implement a retry pattern, as long as you use the IHttpClient and IHttpClientFactory.

But enugh talk, lets code.

STEP 1: THE NUGET PACKAGES

You need (at least) the following NuGet Packages:

  • Polly
  • Microsoft.Extensions.Http.Polly

STEP 2: CONFIGURE SERVICES IN STARTUP.CS

In the services configuration, you need to add a IHttpClientFactory and attach a PolicyHandler to the factory:

//ConfigureServices()  - Startup.cs
services.AddHttpClient("HttpClient").AddPolicyHandler(GetRetryPolicy());

private static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy()
{
  return HttpPolicyExtensions
    // Handle HttpRequestExceptions, 408 and 5xx status codes
    .HandleTransientHttpError()
    // Handle 404 not found
	.OrResult(msg => msg.StatusCode == System.Net.HttpStatusCode.NotFound)
    // Handle 401 Unauthorized
	.OrResult(msg => msg.StatusCode == System.Net.HttpStatusCode.Unauthorized)
    // What to do if any of the above erros occur:
	// Retry 3 times, each time wait 1,2 and 4 seconds before retrying.
	.WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));
}

STEP 3: USE THE IHttpClientFactory IN THE CALLING CLASS

The IHttpClientFactory can be injected using constructor injection. The cool part of the Polly implementation is that your HttpClient code does not contain any special retry-code, just the usual Get or Post calls:

namespace MyCode
{
  public class MyClass
  {
    private readonly IHttpClientFactory _clientFactory;

    public MyClass(IHttpClientFactory clientFactory)
    {
      _clientFactory = clientFactory;
    }

    public async Task<int> PostMessage(string postData)
    {
      var httpClient = _clientFactory.CreateClient("HttpClient");

      using (var content = new StringContent(postData, Encoding.UTF8, "application/json"))
      {
        var result = await httpClient.PostAsync($"{url}", content);
        // The call was a success
        if (result.StatusCode == HttpStatusCode.Accepted)
        {
          return result.StatusCode;
        }
        // The call was not a success, do something
        else
        {
          // Do something
          return result.StatusCode;
        }
      }
    }
  }
}

The httpClient.PostAsync() will retry the post call automatically if any of the conditions described in the GetRetryPolicy() occurs. It will only return after the call is either successful or the retry count is met.

MORE TO READ:


Remove duplicates from XML feed

$
0
0

Apparently XML isn’t dead yet, and today I received a Google Product Feed in the RSS 2.0 XML format. The feed was full of duplicates and my job is to remove them:

<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0" xmlns:g="http://base.google.com/ns/1.0">
    <channel>
        <item>
            <g:id>100</g:id>
            <title>Product 100</title>
            ...
            ...
        </item>
        <item>
            <g:id>100</g:id>
            <title>Product 100</title>
            ...
            ...
        </item>
        <item>
            <g:id>200</g:id>
            <title>Product 200</title>
            ...
            ...
        </item>
        <item>
            <g:id>300</g:id>
            <title>Product 300</title>
            ...
            ...
        </item>
    </channel>
</rss>

As you can see, “Product 100” appears twice.

THE SOLUTION:

A little LINQ can get you far:

using System.Xml;
using System.Xml.Linq;
using System.Linq;

var document = XDocument.Parse(theXMLString);

XNamespace g = "http://base.google.com/ns/1.0";
document.Descendants().Where(node => node.Name == "item");
    .GroupBy(node => node.Element(g+"id").Value)
    .SelectMany(node => node.Skip(1))
    .Remove();

HOW IT WORKS:

  • document.Descendants().Where(node => node.Name == “item”): Get all elements called “item
  • GroupBy(node => node.Element(g+”id”).Value): Group them by the “g:id” element.
  • SelectMany(node => node.Skip(1)): Select every one of them apart from the first one
  • Remove(): Delete all that were selected

MORE TO READ:

Sitecore high memory usage – not always a problem

$
0
0

Sitecore is known to be a real memory hog. Especially the CM server is known to love memory. One of the memory issues that Sitecore faces is the high usage of the Large Object Heap.

Heap Fragmentation

Heap Fragmentation

WHY IS THIS A PROBLEM?

The .net garbage collector will start collecting memory from Generation 0. Objects that cannot be collected is upgraded to Generation 1, which is then collected later. Objects can be further upgraded to Generation 2, assuming that these objects are long-lived and does not need to be collected as often.

The Large Object Heap is where objects larger than 85.000 bytes live. This is mostly strings and arrays, and these objects are rarely garbage collected, and the heap is never compacted (not entirely true as newer .net versions can compact this heap) with can lead to high memory usage.

WHY THIS IS PROBABLY NOT A PROBLEM ANYWAY?

Sitecore is well-known to be a large object heap eater, because Sitecore stores fields and field values as arrays of strings, making every Item a large object. And this is a problem – right?

It definitely can be a problem – but not necessarily:

  • The Large Object heap IS garbage collected, but not very often. It will be garbage collected when the dynamic threshold is met, AND when you run low on memory.
  • .NET will consume the memory available. There is no need for a garbage collection if you are not running low on memory. It can be nerve wrecking to see the memory being low on your server, but this is often not a problem, it is by design.
  • The high fragmentation of your large object heap is a sign that objects are being freed, leaving holes in the heap (so at least your application is not leaking memory). These holes will be filled with new objects if they fit into the hole.
  • Memory is becoming increasingly cheaper, making the business case of memory optimization smaller and smaller. Memory leaks is still a problem, but memory usage is not.

HEY, IT’S A PROBLEM FOR ME. WHAT CAN I DO ABOUT IT?

But let’s say that memory IS a problem – what can you do about it?

TUNE YOUR CACHES:

If memory is a problem, you should not let the cache sizes run free. Set Caching.DisableCacheSizeLimits to false and tune your Sitecore caches:

<setting name="Caching.DisableCacheSizeLimits" value="false" />

I have provided a list of links to cache tuning guides at the bottom of this post.

REUSE DUPLICATED FIELD VALUES:

.NET introduced a string object pool pattern that allows you to reference strings that are duplicated. Strings cannot be changed once created (also called “immutable objects”), and can therefore be referenced using interning. But only if you code it that way.

Sitecore have implemented an interning pattern in the \App_Config\Sitecore\CMS.Core\Sitecore.Interning.config file where fields that often have duplicated values can be stored:

<interning>
  <fieldIdsToIntern>
    <workflowState>{3E431DE1-525E-47A3-B6B0-1CCBEC3A8C98}</workflowState>
    <workflow>{A4F985D9-98B3-4B52-AAAF-4344F6E747C6}</workflow>
    <updatedBy>{BADD9CF9-53E0-4D0C-BCC0-2D784C282F6A}</updatedBy>
    <createdBy>{5DD74568-4D4B-44C1-B513-0AF5F4CDA34F}</createdBy>
    ...
    ...
  </fieldIdsToIntern>
</interning>

You can add your own duplicated fields to the list. Nikolay Mitikov created this SQL to find duplicated values:

WITH DuplicatedFieldValues AS  (
SELECT 
	v.FieldId,	
	FieldDefinitionRow.[Name],
	CONVERT(NVARCHAR(250),v.[Value]) AS [Field Value],
	COUNT(1) AS [Hits]
FROM 
	[VersionedFields] v 
JOIN 
	[Items] FieldDefinitionRow ON FieldDefinitionRow.ID = v.fieldID
WHERE 
	v.FieldId NOT IN 
	(
		/* Fields already interned OOB by Sitecore.Interning.config */			
		'BADD9CF9-53E0-4D0C-BCC0-2D784C282F6A' /* updated by */,
		'5DD74568-4D4B-44C1-B513-0AF5F4CDA34F' /* created by */,
		'52807595-0F8F-4B20-8D2A-CB71D28C6103' /* owner */,
		'3E431DE1-525E-47A3-B6B0-1CCBEC3A8C98' /* workflow state */	
	)
GROUP BY 
	FieldId, [Name], CONVERT(NVARCHAR(250),[Value])
HAVING 
	COUNT(*) > 500 /* How many same field values must be met to be shown */)

SELECT * FROM DuplicatedFieldValues
ORDER BY [Hits] DESC

This can release a lot of memory, especially if you have large text fields.

USE Sitecore.Diagnostics.MemoryMonitorHook TO CLEAR CACHE AND FORCE GARBAGE COLLECTION

The Sitecore.Diagnostics.MemoryMonitorHook monitors your memory usage and can be set to force a garbage collection and a cache clear when the threshold is met:

<hooks>
    <hook type="Sitecore.Diagnostics.MemoryMonitorHook, Sitecore.Kernel">
        <param desc="Threshold">8192MB</param>
        <param desc="Check interval">00:15:00</param>
        <ClearCaches>true</ClearCaches>
        <GarbageCollect>true</GarbageCollect>
        <AdjustLoadFactor>false</AdjustLoadFactor>
    </hook>
</hooks>
  • Threshold: How much memory should be used before doing anything. Set this to 60-80% of your total memory.
  • Check interval: How often should the hook run.
  • ClearCaches: When threshold is met, should the cache be cleared?
  • GarbageCollect: When threshold is met, should Sitecore do a forced garbage collect?
  • AdjustLoadFactor: When threshold is met, should Sitecore attempt to adjust the garbage collection load factor?

MORE TO READ:

 

Sitecore Memory Issues – Every memory optimization trick in the book

$
0
0

My recent post, Sitecore high memory usage – not always a problem, claims that high memory usage is not the same as having a memory problem in Sitecore. But the reason you are reading this blog is because the memory have become a problem – right? So let’s see what you can do about it.

I will not go into the real issue, which probably is that your own application code have a memory leak. This not Sitecore’s fault, although we would like it to be. Instead I will show you the Sitecore settings that will limit the memory consumption of the Sitecore platform itself.

TRICK 1: YOUR EVENT QUEUE IS RUNNING HIGH

This is the #1 issue: Too many events in the event queue. First you need to clean the event queue. Read how to clean the event queue here.

Next you need to manage the event queue size. Use the CleanupEventQueue agent and tune the cleanup so you have no more than 7000 items in the queue at any time:

<agent type="Sitecore.Tasks.CleanupEventQueue, Sitecore.Kernel" method="Run" interval="04:00:00">
  <IntervalToKeep>04:00:00</IntervalToKeep>
  <DaysToKeep>1</DaysToKeep>
</agent>

TRICK 2: DON’T LET THE SITECORE CACHE RUN WILD

Use the Caching.DisableCacheSizeLimits to enforce the cache limits you have painfully configured through the application:

<setting name="Caching.DisableCacheSizeLimits" value="false" />

Also use one of the many Sitecore Cache Tuning Guides online. Even old blog posts will still be useful, as the cache is part of Sitecore basics and have not changed much.

TRICK 3: YOU HAVE TOO MANY UNUSED VERSIONS OF YOUR ITEMS

Items with many versions will eat your memory every time you open the item in the Sitecore Shell. If possible, delete the old versions. Click here to see how to delete old Sitecore item versions.

TRICK 4: DISABLE EXM

Are you using EXM? No? Then disable EXM:

<setting name="EXM.Enabled" value="false" />

TRICK 6: DO NOT CACHE ITEMS WHEN PUBLISHING

You can disable the Publishing database cache. Then the items being published will not be added to the Sitecore database cache.

<setting name="Publishing.DisableDatabaseCaches" value="true"/>

TRICK 7: LOWER THE BATCH SIZE WHEN INDEXING

The Sitecore SOLR indexing can eat your memory. Lowering the batch size have a positive effect on the amount of memory being used when an item is indexed:

<setting name="ContentSearch.IndexUpdate.BatchSize" value="150" />

TRICK 8: DO NOT CACHE WHEN INDEXING

You can disable the index caching. Then the items being retrieved during an index is not cached:

<setting name="ContentSearch.Indexing.DisableDatabaseCaches" value="true" />

TRICK 9: UTILIZE INTERNING

.NET introduced a string object pool pattern that allows you to reference strings that are duplicated. Strings cannot be changed once created (also called “immutable objects”), and can therefore be referenced using interning. Sitecore have implemented interning on fields, but you have to define which fields should be interned. Legend says that there is a performance penalty when interning, but I have not been able to measure any performance decrease.

<interning>
  <fieldIdsToIntern>
    <workflowState>{3E431DE1-525E-47A3-B6B0-1CCBEC3A8C98}</workflowState>
    <workflow>{A4F985D9-98B3-4B52-AAAF-4344F6E747C6}</workflow>
    <updatedBy>{BADD9CF9-53E0-4D0C-BCC0-2D784C282F6A}</updatedBy>
    <createdBy>{5DD74568-4D4B-44C1-B513-0AF5F4CDA34F}</createdBy>
    ...
    ...
  </fieldIdsToIntern>
</interning>

TRICK 10: FORCE GARBAGE COLLECTION WHEN MEMORY IS LOW

The old peeing in your pants trick is also implemented by Sitecore. You need to enable counters first:

<setting name="Counters.Enabled" value="true"/>

Then you can allow the MemoryMonitorHook to force a garbage collection when memory is running low:

<hooks>
    <hook type="Sitecore.Diagnostics.MemoryMonitorHook, Sitecore.Kernel">
        <param desc="Threshold">8192MB</param>
        <param desc="Check interval">00:15:00</param>
        <ClearCaches>true</ClearCaches>
        <GarbageCollect>true</GarbageCollect>
        <AdjustLoadFactor>false</AdjustLoadFactor>
    </hook>
</hooks>

I guess that’s it. If you know of a memory trick for Sitecore, please let me know.

MORE TO READ:

 

C# Azure TelemetryClient will leak memory if not implemented as a singleton

$
0
0

I noticed that my classic .net web application would leak memory after I implemented metrics for some background tasks.

Memory usage of web application

Memory usage of web application

Further investigation showed that my MetricAggregationManager would not release its memory.

Object was not garbage collected

Object was not garbage collected

Since one of the major changes was the implementation of a TelemetryClient, and since the memory not being released was from the Microsoft.ApplicationInsights.Metrics namespace, I concluded that the problem lies within the creation of the TelemetryClient:

using System;
using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.Metrics;

namespace MyCode
{
  public class BaseProcessor
  {
    private readonly TelemetryClient _telemetryClient;
    
    private BaseProcessor()
    {
      string instrumentationKey = "somekey"
      var telemetryConfiguration = new TelemetryConfiguration { InstrumentationKey = instrumentationKey };
      // This is a no-go. I should not create a new instance for every BaseProcessor
      _telemetryClient = new TelemetryClient(telemetryConfiguration);
    }
  }
}

The code above will create a new TelemetryClient for each creation of my base class. The TelemetryClient will collect metrics and store those in memory until either a set time or number of metrics are met, and then dump the metrics to Application Insights.

So when the BaseClass is disposed, TelemetryClient is not, leaving memory to hang, and thus a memory leak is in effect.

HOW TO SOLVE IT?

The solution is simple. All you need to do is to create a singleton pattern for your TelemetryClient. Having only one instance will allow the client to collect and send metrics in peace. Your code will be much faster (it takes a millisecond or so to create a TelemetryClient) and you will not have any memory leaks.

USE DEPENDENCY INJECTION:

In .NET Core you can add the TelemetryClient to the service collection:

private static void ConfigureServices(IServiceCollection services)
{
  // Add Application Insights
  var telemetryConfiguration = TelemetryConfiguration.CreateDefault();
  telemetryConfiguration.InstrumentationKey = "somekey"
  var telemetryClient = new TelemetryClient(telemetryConfiguration);
  services.AddSingleton(telemetryClient);
}

And then reference it using constructor injection:

using System;
using System.Runtime.Serialization;
using Microsoft.ApplicationInsights;
using Microsoft.AspNetCore.Mvc;

namespace MyCode
{
  [ApiController]
  [Route("/api/[controller]")]
  [Produces("application/json")]
  public class MyController : ControllerBase
  {
    private readonly TelemetryClient _telemetryClient;

    public MyController(TelemetryClient telemetryClient)
    {
      _telemetryClient = telemetryClient;
    }
  }
}

USE A STATIC VARIABLE:

If you do not have access to a DI framework, you could also just create a static variable:

using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.Extensibility;
using System.Collections.Generic;

namespace MyCode
{
  public static class TelemetryFactory
  {
    private static TelemetryClient _telemetryClient;

    public static TelemetryClient GetTelemetryClient()
    {
      if (_telemetryClients == null)
      {
        string instrumentationKey = "somekey";
        var telemetryConfiguration = new TelemetryConfiguration { InstrumentationKey = instrumentationKey };
        _telemetryClient = new TelemetryClient(telemetryConfiguration);
      }

      return _telemetryClient;
    }
  }
}

And then reference the static variable instead:

using System;
using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.Metrics;

namespace MyCode
{
  public class BaseProcessor
  {
    private readonly TelemetryClient _telemetryClient;
    
    private BaseProcessor()
    {
      _telemetryClient = TelemetryFactory.GetTelemetryClient();
    }
  }
}

MORE TO READ:

 

Manipulating XML Google Merchant Data using C# and LINQ

$
0
0

Receiving a Google Merchant Data feed (also known as a Google Product Feed) can be fairly easily manipulated on import time using a little C# and LINQ.

The feed is basically a XML RSS 2.0 feed with some added properties using the namespace xmlns:g=”http://base.google.com/ns/1.0.

These feeds often comes from older systems and data is created by busy merchants, so data can be relatively dirty, and a cleanup is required before you add them to your product database.

The feed could look like this:

<?xml version="1.0" encoding="utf-8" ?>
<rss version="2.0" xmlns:g="http://base.google.com/ns/1.0">
    <channel>
        <title>Google product feed</title>
        <link href="https://pentia.dk" rel="alternate" type="text/html"/>
        <description>Google product feed</description>
        <item>
            <g:id><![CDATA[1123432]]></g:id>
            <title><![CDATA[Some product]]></g:title>
            <link><![CDATA[https://pentia.dk]]></g:link>
            <g:description><![CDATA[description]]></g:description>
            <g:gtin><![CDATA[5712750043243446]]></g:gtin>
            <g:mpn><![CDATA[34432-00]]></g:mpn>
            <g:image_link><![CDATA[https://pentia.dk/someimage.jpg]]></g:image_link>
            <g:product_type><![CDATA[Home &gt; Dresses &gt; Maxi Dresses]]></g:product_type>
            <g:condition><![CDATA[new]]></g:condition>
            <g:availability><![CDATA[in stock]]></g:availability>
            <g:price><![CDATA[15.00 USD]]></g:price>
            <g:sale_price><![CDATA[10.00 USD]]></g:sale_price>
        </item>
        ...
        ...
    </channel>
</rss>

See the full specification in the Google Merchant Center help.

Sometimes the feed would contain content that you does not need, and a little XML manipulation is required.

But first thing first:

STEP 1: GET THE XML FEED AND CONVERT IT INTO AN XML DOCUMENT

using System;
using System.Net;
using System.Net.Http;
using System.Xml;
using System.Xml.Linq;
using System.Linq;
using System.Dynamic;

private static HttpClient _httpClient = new HttpClient();

public static async Task<string> GetFeed(string url)
{
  using (var result = await _httpClient.GetAsync($"{url}"))
  {
    string content = await result.Content.ReadAsStringAsync();
    return content;
  }
}

public static void Run()
{
  // Get the RSS 2.0 XML data
  string feedData = GetData("https://url/thefeed.xml").Result;

  // Convert the data into an XDocument
  var document = XDocument.Parse(feedData);
  // Speficy the Google namespace
  XNamespace g = "http://base.google.com/ns/1.0";
  // Get a list of all "item" nodes
  var items = document.Descendants().Where(node =&amp;gt; node.Name == "item");
    
  // Now we are ready to manipulate
  // ...
  // ...
}

NOW TO THE MANIPULATIONS:

EXAMPLE 1: Remove duplicates – all products with the same ID is removed:

items.GroupBy(node => node.Element(g+"id").Value)
  .SelectMany(node => node.Skip(1))
  .Remove();

EXAMPLE 2: Remove all products out of stock:

items = document.Descendants()
  .Where(node => node.Name == "item" 
         && node.Descendants()
         .Any(desc => desc.Name == g + "availability" 
              && desc.Value == "out of stock"));
items.Remove();

EXAMPLE 3: Remove adverts not on sale (all adverts that do not have a g:sale_price node)

items = document.Descendants()
  .Where(node => node.Name == "item" 
         && node.Descendants()
         .Any(desc => desc.Name == g + "sale_price" 
         && desc.Value.Trim() == string.Empty));
items.Remove();

EXAMPLE 4: ADD TRACKING PARAMETERS TO URL’S (adding query string parameters to the URL)

var items = document.Descendants().Where(node => node.Name == "item");
foreach (var item in items)
{
  string url = item.Element("link").Value;
  if (url.Contains("?"))
    item.Element("link").ReplaceNodes(new XCData(url + "&" + "utm_source=s&utm_medium=m&utm_campaign=c"));
  else  
    item.Element("link").ReplaceNodes(new XCData(url + "?" + "utm_source=s&utm_medium=m&utm_campaign=c"));
}

EXAMPLE 5: CHANGE THE TITLE (for example, if the feed contains used products, you might want to add the word “used” to the title

var items = document.Descendants().Where(node => node.Name == "item");
foreach (var item in items)
{
  var title = "USED " + item.Element("title").Value;
  item.Element("title").ReplaceNodes(title);
}

…AND THE EXOTIC EXAMPLE: COMBINE ALL PRODUCTS IF THEY BELONG TO A PRODUCT_TYPE THAT CONTAIN LESS THAN 2 PRODUCTS

foreach(var group in items.GroupBy(node => node.Element(g+"product_type").Value))
{
  if (group.Count() <= 2)
  {
    foreach (var advert in group)
    {
      advert.Element(g+"product_type").ReplaceNodes(new XCData("Other"));
    }
  }
}

Finally you can grab the manipulated document and do what you need to do:

// Grab converted content
string convertedFeedData = document.ToString();

I hope this gives some examples on how to do much with less code.

MORE TO READ:

Viewing all 285 articles
Browse latest View live