////
    Licensed to the Apache Software Foundation (ASF) under one or more
    contributor license agreements.  See the NOTICE file distributed with
    this work for additional information regarding copyright ownership.
    The ASF licenses this file to You under the Apache License, Version 2.0
    (the "License"); you may not use this file except in compliance with
    the License.  You may obtain a copy of the License at

         http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
////

= Garbage-free logging

include::partial$manual/garbagefree-intro.adoc[]

include::partial$premature-optimization.adoc[]

The act of logging is an interplay between the logging API (i.e., Log4j API) where the programmer publishes logs and a logging implementation (i.e., Log4j Core) where published logs get consumed; filtered, enriched, encoded, and written to files, databases, network sockets, etc.
Both parties contain different features with different memory allocation characteristics.
To achieve an end-to-end garbage-free logging system, they need to work hand in hand.
Hence, we will discuss both:

. <<Config>>
. <<api>>

[NOTE]
====
Garbage-free logging is currently implemented for Log4j API and its reference implementation, Log4j Core.
If you use another setup (e.g., a different logging API or implementation) this promise might not hold.
====

[#quick]
== Quick start

If you want to have a garbage-free Log4j setup, but don't want to spend time with the associated details, you can quickly get started with the following instructions:

. Set the following system properties to `true`:
** <<log4j.enableThreadlocals,`log4j.enableThreadlocals`>>
** <<log4j.threadContext.map.garbageFree,`log4j.threadContext.map.garbageFree`>>
. Use garbage-free
** <<Layouts>>
** <<Appenders>>
** <<Filters>>

This should be sufficient for a majority of use cases.
If not for yours, keep on reading.

[#Config]
== Log4j Core configuration

In order to have a garbage-free Log4j Core, you need to

* <<core-properties,configure it using properties>>,
* and employ garbage-free <<Layouts,layouts>>, <<Appenders,appenders>>, and <<Filters,filters>>.

[#core-properties]
=== Properties

Garbage-free logging can be configured for Log4j Core using properties listed below.
(See xref:manual/configuration.adoc[] on details how you can set these properties.)

include::partial$manual/systemproperties/properties-meta.adoc[leveloffset=+2]

include::partial$manual/systemproperties/properties-garbage-collection.adoc[leveloffset=+2]

include::partial$manual/systemproperties/properties-thread-context-core.adoc[leveloffset=+2,tag=gcfree]

[#Layouts]
=== Layouts

The following xref:manual/layouts.adoc[layouts] can be configured to run garbage-free during steady-state logging.
To understand which configuration knobs exhibit what kind of allocation behaviour, see their dedicated pages.

// Maintain the alphabetical ordering while making changes, please!
* xref:manual/json-template-layout.adoc#faq-garbage-free[`JsonTemplateLayout`]
* xref:manual/pattern-layout.adoc#garbage-free[`PatternLayout`]

.Implementation notes
[%collapsible]
====
Garbage-free xref:manual/layouts.adoc[layouts] need to implement the `Encoder<LogEvent>` interface.
link:../javadoc/log4j-core/org/apache/logging/log4j/core/layout/StringBuilderEncoder.html[`StringBuilderEncoder`] helps with encoding text to bytes in a garbage-free manner.
====

[#Appenders]
=== Appenders

The following xref:manual/appenders.adoc[appenders] are garbage-free during steady-state logging:

// Maintain the alphabetical ordering while making changes, please!
* xref:manual/appenders.adoc#ConsoleAppender[`ConsoleAppender`]
* xref:manual/appenders/file.adoc#FileAppender[`FileAppender`]
* xref:manual/appenders/file.adoc#MemoryMappedFileAppender[`MemoryMappedFileAppender`]
* xref:manual/appenders/file.adoc#RandomAccessFileAppender[`RandomAccessFileAppender`]
* xref:manual/appenders/rolling-file.adoc#RollingFileAppender[`RollingFileAppender`] (except during rollover)
* xref:manual/appenders/rolling-file.adoc#RollingRandomAccessFileAppender[`RollingRandomAccessFileAppender`] (except during rollover)

Any other appender not shared in the above list (including xref:manual/appenders/delegating.adoc#AsyncAppender[`AsyncAppender`]) is not garbage-free.

.Implementation notes
[%collapsible]
====
Garbage-free xref:manual/appenders.adoc[appenders] need to provide their xref:manual/layouts.adoc[layout] with a `ByteBufferDestination` implementation that the layout can directly write into.

[NOTE]
=====
`AbstractOutputStreamAppender` has been modified to make the following appenders garbage-free:

* `ConsoleAppender`
* `(Rolling)FileAppender`
* `(Rolling)RandomAccessFileAppender`
* `MemoryMappedFileAppender`

An effort has been made to minimize impact on custom appenders that extend `AbstractOutputStreamAppender`, but it is impossible to guarantee that changing the superclass will not impact any and all subclasses.
Custom appenders that extend `AbstractOutputStreamAppender` should verify that they still function correctly.
In case there is a problem, the <<log4j.gc.enableDirectEncoders,`log4j.gc.enableDirectEncoders`>> configuration property can be set to `false` to revert to the pre-Log4j 2.6 behaviour.
=====
====

[#Filters]
=== Filters

The following xref:manual/filters.adoc[filters] are garbage-free during steady-state logging:

// Maintain the alphabetical ordering while making changes, please!
* xref:manual/filters.adoc#CompositeFilter[`CompositeFilter`] (adding and removing element filters creates temporary objects for thread safety)
* xref:manual/filters.adoc#DynamicThresholdFilter[`DynamicThresholdFilter`]
* xref:manual/filters.adoc#LevelRangeFilter[`LevelRangeFilter`] (garbage-free since `2.8`)
* xref:manual/filters.adoc#MapFilter[`MapFilter`] (garbage-free since `2.8`)
* xref:manual/filters.adoc#MarkerFilter[`MarkerFilter`] (garbage-free since `2.8`)
* xref:manual/filters.adoc#StructuredDataFilter[`StructuredDataFilter`] (garbage-free since `2.8`)
* xref:manual/filters.adoc#ThreadContextMapFilter[`ThreadContextMapFilter]` (garbage-free since `2.8`)
* xref:manual/filters.adoc#ThresholdFilter[`ThresholdFilter`] (garbage-free since `2.8`)
* xref:manual/filters.adoc#TimeFilter[`TimeFilter`] (garbage-free since `2.8` except when range must be recalculated once per day)

Any other filter not shared in the above list is not garbage-free.

[#recyclers]
=== Recyclers

Log4j contains the _recycler_ abstraction that models the buffer-and-reuse strategy used to operate without any extra memory allocations at steady state.
Recyclers implement the `RecyclerFactoryProvider` interface in `log4j-kit` (a dependency of `log4j-core`) and provide themselves as a `ServiceLoader`.
Hence, you can easily implement custom recyclers and inject them using the `ServiceLoader` mechanism.

Log4j chooses a recycler using the following procedure:

. Using `ServiceLoader` mechanism, loads all available `RecyclerFactoryProvider` implementations
. Sorts providers (the lower `order` value will come first)
. If <<#log4j.recycler.factory>> is provided, the first provider whose name matching with this property value will be used.
Otherwise, the first provider will be used.

The predefined recycler factory providers are as follows:

[#recyclers-dummy]
==== Dummy recycler

[cols="1h,5"]
|===
| Module
| `log4j-kit`

| Name
| `dummy`

| Order
| 900
|===

Does not recycle anything; all instances are freshly created.
(Not recommended for production!
Intended as a test utility.)

[#recyclers-threadLocal]
==== Thread-local recycler

[cols="1h,5"]
|===
| Module
| xref:components.adoc#log4j-kit[`log4j-kit`]

| Name
| `threadLocal`

| Order
| `Integer.MAX_VALUE`, if `javax.servlet.Servlet` or `jakarta.servlet.Servlet` is in the classpath; +
700, otherwise
|===

Pools objects in a fixed-size queue stored in a `ThreadLocal`.
If pool runs out of capacity (see <<#log4j.recycler.capacity>>), it will start creating fresh instances for new requests.

Note that it is effectively disabled for servlet environments.

[#recyclers-queue]
==== Queueing recycler

[cols="1h,5"]
|===
| Module
| xref:components.adoc#log4j-kit[`log4j-kit`]

| Name
| `queue`

| Order
| 800
|===

Pools objects in a fixed-size `ArrayBlockingQueue` queue.
If pool runs out of capacity (see <<log4j.recycler.capacity>>), it will start creating fresh instances for new requests.

[#recyclers-jctools]
==== JCTools MPMC recycler

[cols="1h,5"]
|===
| Module
| xref:components.adoc#log4j-jctools[`log4j-jctools`]

| Name
| `jctools-mpmc`

| Order
| 600
|===

Pools objects in a fixed-size JCTools MPMC queue.
If pool runs out of capacity (see <<log4j.recycler.capacity>>), it will start creating fresh instances for new requests.

[#recyclers-conversant]
==== Conversant Disruptor recycler

[cols="1h,5"]
|===
| Module
| xref:components.adoc#log4j-conversant[`log4j-conversant`]

| Name
| `conversant-disruptor`

| Order
| 600
|===

Pools objects in a fixed-size ring buffer.
If pool runs out of capacity (see <<log4j.recycler.capacity>>), a spin policy is applied (see <<log4j.recycler.conversant.spinPolicy>>).

[NOTE]
====
The capacity of this recycler **must** be at least 8 and a power of 2.
====

[#core-limitations]
=== Limitations

There are certain caveats associated with the configuration of garbage-free logging:

[#core-limitation-properties]
Property substitutions::

Some xref:manual/configuration.adoc#property-substitution[property substitutions] (e.g., ones using xref:manual/lookups.adoc#DateLookup[Date Lookup]) might result in temporary objects being created during steady-state logging.

[#core-limitation-async-logger-wait-strategy]
Asynchronous logger wait strategies::

As of version `2.18.0`, the default xref:manual/async.adoc[asynchronous logger] wait strategy (i.e., `Timeout`) is garbage-free while running against both LMAX Disruptor 3 and 4.
See
xref:manual/async.adoc#log4j.async.logger.waitStrategy.type[`log4j.async.logger.waitStrategy.type`]
for details on predefined wait strategies.

[#api]
== Log4j API usage

xref:manual/api.adoc[Log4j API] contains several features to facilitate garbage-free logging:

[#api-vararg]
=== Parameterized message arguments

The `Logger` interface contains methods for parameterized messages up to 10 arguments.
Logging more than 10 parameters creates https://docs.oracle.com/javase/8/docs/technotes/guides/language/varargs.html[vararg arrays].

[#api-encode-custom-object]
=== Encoding custom objects

When a message parameter contains an unknown type by the layout, it will encode by calling `toString()` on these objects.
Most objects don't have garbage-free `toString()` methods.
Objects themselves can implement their own garbage-free encoders by either extending from
https://docs.oracle.com/en/java/javase/{java-target-version}/docs/api/java.base/java/lang/CharSequence.html[Java's `CharSequence`]
or
{log4j2-url}/javadoc/log4j-api/org/apache/logging/log4j/util/StringBuilderFormattable.html[Log4j's `StringBuilderFormattable`].

[#codeImpact]
=== Avoiding autoboxing

We made an effort to make logging garbage-free without requiring code changes in existing applications, but there is one area where this was not possible.
When logging primitive values (i.e. int, double, boolean, etc.) the JVM autoboxes these primitive values to their Object wrapper equivalents, creating garbage.

Log4j provides an `Unbox` utility to prevent autoboxing of primitive parameters.
This utility contains a thread-local pool of reused
`StringBuilder`s. The `Unbox.box(primitive)` methods write directly into a StringBuilder, and the resulting text will be copied into the final log message text without creating temporary objects.

[source,java]
----
import static org.apache.logging.log4j.util.Unbox.box;

LOGGER.debug("Prevent primitive autoboxing {} {}", box(10L), box(2.6d));
----

This utility contains a `ThreadLocal` pool of reused ``StringBuilder``s.
The pool size is configured by <<log4j2.unboxRingbufferSize,the `log4j2.unboxRingbufferSize` system property>>.
The `Unbox.box(primitive)` methods write directly into a `StringBuilder`, and the resulting text will be copied into the final log message text without creating temporary objects.

[#api-limitations]
=== Limitations

Not all Log4j API feature set is garbage-free, specifically:

* The `ThreadContext` map (aka. MDC) is not garbage-free by default, but can be configured to be garbage-free by setting the <<log4j.threadContext.map.garbageFree>> configuration property to `true`.
* The `ThreadContext` stack (aka. NDC) is not garbage-free.
* {log4j2-url}/manual/scoped-context.html[Scoped Context] is not garbage-free.
* Logging very large messages (i.e., more than <<log4j.gc.maxReusableMsgSize>> characters, which defaults to 518), when all loggers are xref:manual/async.adoc[asynchronous loggers], will cause the internal `StringBuilder` in the
`RingBuffer` to be trimmed back to their configured maximum size.
* Logging messages containing `$\{variable}` substitutions creates temporary objects.
* Logging a lambda as a parameter:
+
[source,java]
----
LOGGER.info("lambda value is {}", () -> callExpensiveMethod());
----
+
creates a vararg array.
Logging a lambda expression by itself:
+
[source,java]
----
LOGGER.debug(() -> callExpensiveMethod());
----
+
is garbage-free.
* The {log4j2-url}/javadoc/log4j-api/org/apache/logging/log4j/Logger.html#traceEntry()[`traceEntry()`] and {log4j2-url}/javadoc/log4j-api/org/apache/logging/log4j/Logger.html#traceExit()[`traceExit()`] methods create temporary objects.
* Time calculations are not garbage-free when the `log4j2.usePreciseClock` system property (defaults to `false`) is set to `true`.
