Performance profiling is the process of reviewing and debugging applications for bottlenecks and inefficiencies to identify areas for improvement.

To profile, we utilise tools that provide us with deep telemetry data to point to these areas of improvement. The golden standard for these tools is paid subscriptions e.g. New Relic and Backfire.io. On the open source side, popular tools such as Xdebug and SPX are great but lack the features needed for profiling in production.

Open source options are outclassed by paid subscription-based services because of the effort and deep technical knowledge required to implement them. But there is hope...

eBPF (Extended Berkeley Packet Filter) lets programmers load and execute lightweight programs within the Linux kernel. This is a big deal because it enables us to easily observe and instrument our applications in ways that were incredibly difficult to do in the past.

This talk outlines the journey we have undertaken to build an open-source telemetry system on top of eBPF, Rust, and Go for instrumenting the PHP applications which run on our hosting platform, Skpr.

In this talk, I will cover the following topics:
- The current state of monitoring and tracing
- What is eBPF?
- How to capture tracing data with eBPF
- What is PHP's Observer API?
- How to write a PECL extension with Rust
- The decisions we made when designing our tracing implementation