LogoLogo
  • Welcome
  • Where to Start
    • Introduction
    • list of Operations
    • Bantu Stack
  • Tutorials
    • Create Account
    • Send and Receive Payments
    • Follow Received Payments
    • Securing Web-based Projects
  • Issue Assets
    • Overview
    • Anatomy of an Asset
    • Issue an Asset
    • Publish Information About an Asset
    • Control Access to an Asset
  • Building Apps
    • Overview
    • Project Setup
    • Key Management Basics
    • Create a Basic Wallet
    • Make XBN Payments
    • Handle Custom Assets
  • Run a Core Node
    • index
    • running-node
    • network-upgrades
    • tier-1-orgs
    • installation
    • prerequisites
    • configuring
    • publishing history archives
    • commands
    • monitoring
  • Run API Server
    • prerequisites
    • quickstart
    • index
    • installing
    • monitoring
    • Running
    • configuring
  • Software and SDKs
    • index
  • Glossary
    • scp
    • Claimable Balance
    • XDR
    • Assets
    • BUDS
    • Inflation
    • Miscellaneous Core Objects
    • Testnet
    • Accounts
    • Network Passphrase
    • Ledger
    • Versioning
    • Sponsored Reserves
    • Operations
    • Decentralized Exchange
    • Fees
    • XBN Supply
    • Fee Bumps
    • Channels
    • Transactions
    • Minimum Balance
    • Multisig
  • Docs
    • Index
  • API
    • Introduction
      • Index
      • Response Format
      • Streaming
      • Rate Limiting
      • XDR
      • Pagination
        • Index
        • Page Arguments
    • Resources
      • Untitled
      • Overview
      • Ledgers
        • index
        • Object
        • Single
        • Transactions
        • Operations
        • Payments
        • Effects
        • List
      • Transactions
        • Index
        • Object
        • Single
        • Operations
        • Effects
        • List
      • Operations
        • Index
        • Object
          • Index
          • Create Account
          • Payment
          • Path Payment Strict Send
          • Path Payment Strict Receive
          • Sell Offer
          • Buy Offer
          • Passive Sell Offer
          • Set Options
          • Change Trust
          • Allow Trust
          • Account Merge
          • Manage Data
          • Bump Sequence
          • Create Claimable Balance
          • Claim Claimable Balance
          • Begin Sponsoring Future Reserves
          • End Sponsoring Future Reserves
          • Revoke Sponsorship
        • Single
        • Effects
        • List
        • List Payments
      • Effects
        • Index
        • Types
        • List
      • Accounts
        • Index
        • Object
        • Transactions
        • List
        • Single
        • Operations
        • Payments
        • Effects
        • Offers
        • Trades
        • Data
      • Offers
        • Index
        • Object
        • Single
        • List
      • Trades
        • Index
        • Object
        • List
      • Assets
        • Index
        • Object
        • List
      • Claimable Balances
        • Index
        • Object
        • Single
        • List
    • Aggregations
      • Index
      • Order Books
        • Index
        • object
        • Single
      • Paths
        • Index
        • Object
        • Strict Receive
        • Strict Send
      • Trade Aggregations
        • Index
        • Object
        • List
      • Fee Stats
        • Index
        • Object
        • Single
    • Errors
      • Index
      • Response
      • HTTP Status Codes
        • Index
        • Standard
        • Expansion Specific
          • Index
          • Transaction Failed
          • Transaction Malformed
          • Before History
          • Stale History
          • Timeout
      • Result Codes
        • Index
        • Transactions
        • Operations
        • Operation Specific
          • Index
          • Create Account
          • Payment
          • Path Payment Strict Receive
          • Path Payment Strict Send
          • Manage Sell Offer
          • Manage Buy Offer
          • Create Passive Sell Offer
          • Set Options
          • Change Trust
          • Allow Trust
          • Account Merge
          • Manage Data
          • Bump Sequence
Powered by GitBook
On this page
  • Starting HTTP request
  • Finished HTTP request
  • Metrics
  • Alerts
  • I'm Stuck! Help!

Was this helpful?

  1. Run API Server

monitoring

PreviousinstallingNextRunning

Last updated 4 years ago

Was this helpful?

import { CodeExample } from "components/CodeExample";

To ensure that your instance of Horizon is performing correctly, we encourage you to monitor it, and provide both logs and metrics to do so.

Horizon will output logs to standard out. Information about what requests are coming in will be reported, but more importantly, warnings or errors will also be emitted by default. A correctly running Horizon instance will not output any warning or error log entries.

Metrics are collected while a Horizon process is running and they are exposed at the /metrics path. You can see an example at .

Below we present a few standard log entries with associated fields. You can use them to build metrics and alerts. Please note that these represent Horizon app metrics only. You should also monitor your hardware metrics like CPU or RAM Utilization.

Starting HTTP request

Key

Value

msg

Starting request

client_name

Value of X-Client-Name HTTP header representing client name

client_version

Value of X-Client-Version HTTP header representing client version

app_name

Value of X-App-Name HTTP header representing app name

app_version

Value of X-App-Version HTTP header representing app version

forwarded_ip

First value of X-Forwarded-For header

host

Value of Host header

ip

IP of a client sending HTTP request

ip_port

IP and port of a client sending HTTP request

method

HTTP method (GET, POST, ...)

path

Full request path, including query string (ex. /transactions?order=desc)

streaming

Boolean, true if request is a streaming request

referer

Value of Referer header

req

Random value that uniquely identifies a request, attached to all logs within this HTTP request

Finished HTTP request

Key

Value

msg

Finished request

bytes

Number of response bytes sent

client_name

Value of X-Client-Name HTTP header representing client name

client_version

Value of X-Client-Version HTTP header representing client version

app_name

Value of X-App-Name HTTP header representing app name

app_version

Value of X-App-Version HTTP header representing app version

duration

Duration of request in seconds

forwarded_ip

First value of X-Forwarded-For header

host

Value of Host header

ip

IP of a client sending HTTP request

ip_port

IP and port of a client sending HTTP request

method

HTTP method (GET, POST, ...)

path

Full request path, including query string (ex. /transactions?order=desc)

route

Route pattern without query string (ex. /accounts/{id})

status

HTTP status code (ex. 200)

streaming

Boolean, true if request is a streaming request

referer

Value of Referer header

req

Random value that uniquely identifies a request, attached to all logs within this HTTP request

Metrics

Using the entries above you can build metrics that will help understand performance of a given Horizon node. For example:

  • Number of requests per minute.

  • Number of requests per route (the most popular routes).

  • Average response time per route.

  • Maximum response time for non-streaming requests.

  • Number of streaming vs. non-streaming requests.

  • Number of rate-limited requests.

  • List of rate-limited IPs.

  • Unique IPs.

  • The most popular SDKs/apps sending requests to a given Horizon node.

  • Average ingestion time of a ledger.

  • Average ingestion time of a transaction.

Alerts

Below are example alerts with potential causes and solutions. Feel free to add more alerts using your metrics:

Alert

Cause

Solution

Spike in number of requests

Potential DoS attack

Lower rate-limiting threshold

Large number of rate-limited requests

Rate-limiting threshold too low

Increase rate-limiting threshold

Ingestion is slow

Horizon server spec too low

Increase hardware spec

Spike in average response time of a single route

Possible bug in a code responsible for rendering a route

Report an issue in Horizon repository.

I'm Stuck! Help!

If any of the above steps don't work or you are otherwise prevented from correctly setting up Horizon, please join our community and let us know. Either or to ask for help.

https://horizon-testnet.stellar.org/metrics
post a question at our Stack Exchange
chat with us on Keybase in #dev_discussion