Skip to content

feat: outlook cache #21072

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 83 commits into
base: main
Choose a base branch
from

Conversation

vijayraghav-io
Copy link
Contributor

@vijayraghav-io vijayraghav-io commented May 2, 2025

What does this PR do?

This PR implements the functionality to -

  • Subscribe for notifications of - create, update or delete events in Microsoft Outlook Calendars (These calendars are pre-connected by team members with their Cal.com accounts, and are mapped to the cal.com events)
  • Receive notifications through a web-hook
  • Validates & filters these notifications and computes the set of outlook calendars (having unique externalIDs) for which the calendar-cache needs invalidation( or update)
  • fetchAvailabilityAndSetCache() is called for above outlook calendars to update the calendar-cache.
  • watch and unwatch functionality is implemented through Microsoft Graph APIs /subscribe , so that the existing cron job can start to subscribe or unsubscribe based on conditions (to avoid duplicate subscriptions).
  • With the calendar-cache available for outlook calendars ,the latest availabilities are shown in real time on public event page with low latency as Graph APIs are not called every time.

Key points:
Visit the team public page with cal.cache query parameter set to true, to see the Cache effect or see [Cache Hit] in logs for office365_calendar.
Example : http://localhost:3000/team/devs/teamevent?overlayCalendar=true&layout=month_view&cal.cache=true

In this PR, the startWatchingCalendarInMicrosoft() calls /subscription endpoint targeting specific calendars. This avoids subscribing to overly broad resources (e.g., me/events for all calendar events). Instead, target specific calendars or folders (e.g., me/calendars/{calendar_id}/events) to reduce notification volume.
Screenshot 2025-05-04 at 11 02 29 PM

Next Steps or Improvements for scaling
Given the volume of usage, if we have very high number of teams, team members, team events, simultaneously updating events in outlook - the volume of subscriptions received by the web-hook can be overwhelming even with load balancers.

We can consider Asynchronous Processing: Process notifications asynchronously to avoid blocking the webhook endpoint. For example, queue incoming notifications (using a cloud message queuing system like AWS SQS or RabbitMQ or Azure Queue Storage) and process them in the background with worker nodes. This ensures the endpoint remains responsive under high load.

Also we can consider using Azure Event Hubs or Azure Event Grid as alternative delivery channels. These services are designed for high-throughput event streaming and can buffer notifications, reducing the load on our application.

Video Demo

This demo shows - on creation of an event in outlook calendar the CalendarCache is updated through the webhook
Cron job was triggered manually through api - http://localhost:3000/api/calendar-cache/cron?apiKey=

https://www.loom.com/share/468bf851a93d49849b2dab27fe751efd?sid=458f9244-e4fe-4632-897d-b93835535ff6

The second demo shows the effect of caching on public team event page-

https://www.loom.com/share/567c3d5af03e478e97a44299e9bcc8da?sid=9254a7e4-a28b-4a49-b17e-2890489fcf84

Mandatory Tasks (DO NOT REMOVE)

  • I have self-reviewed the code (A decent size PR without self-review might be rejected).
  • I have updated the developer docs in /docs if this PR makes changes that would require a documentation change. If N/A, write N/A here and check the checkbox.
  • I confirm automated tests are in place that prove my fix is effective or that my feature works.

How should this be tested?

  • Are there environment variables that should be set?
    MICROSOFT_WEBHOOK_TOKEN - This token is sent with subscription requests through /subscriptions Graph API.
    The notifications received by webhook are verified with this token
  • What are the minimal test data to have?
    Enable calendar-cache feature flag for the team.
    Create a team event with hosts having their outlook calendars connected
  • What is expected (happy path) to have (input and output)?
    At any given point of time Calendar-Cache in cal.com has the latest busy slots in sync with changes in outlook calendar busy times or events.
    Update or create or delete event in Microsoft outlook calendar , there should be instant update in cal.com calendar-cache
    On visiting the team event public page with &cal.cahce=true, latest available slots are displayed with low latency
  • Any other important info that could help to test that PR
    To test locally use port forwarding providers

Summary by mrge

Added caching and webhook support for Outlook (Office365) calendar availability to improve performance and enable real-time updates.

  • New Features
    • Implemented webhook endpoint to receive Outlook calendar change notifications and refresh cache.
    • Added cache layer for Outlook calendar availability queries.
    • Updated database schema to store Outlook subscription info.

@vijayraghav-io vijayraghav-io requested a review from a team as a code owner May 2, 2025 05:56
Copy link

vercel bot commented May 2, 2025

@vijayraghav-io is attempting to deploy a commit to the cal Team on Vercel.

A member of the Team first needs to authorize it.

@graphite-app graphite-app bot added the community Created by Linear-GitHub Sync label May 2, 2025
@graphite-app graphite-app bot requested a review from a team May 2, 2025 05:56
@github-actions github-actions bot added $500 calendar-apps area: calendar, google calendar, outlook, lark, microsoft 365, apple calendar teams area: teams, round robin, collective, managed event-types ✨ feature New feature or request 💎 Bounty A bounty on Algora.io ❗️ migrations contains migration files ❗️ .env changes contains changes to env variables labels May 2, 2025
Copy link

graphite-app bot commented May 2, 2025

Graphite Automations

"Add consumer team as reviewer" took an action on this PR • (05/02/25)

1 reviewer was added to this PR based on Keith Williams's automation.

"Add community label" took an action on this PR • (05/02/25)

1 label was added to this PR based on Keith Williams's automation.

"Add foundation team as reviewer" took an action on this PR • (05/16/25)

1 reviewer was added to this PR based on Keith Williams's automation.

@vijayraghav-io vijayraghav-io requested a review from a team as a code owner May 2, 2025 06:08
Comment on lines +168 to +176
if (!isDelegated) {
const user = await this.fetcher("/me");
const userResponseBody = await handleErrorsJson<User>(user);
this.azureUserId = userResponseBody.userPrincipalName ?? undefined;
if (!this.azureUserId) {
throw new Error("UserPrincipalName is missing for non-delegated user");
}
return this.azureUserId;
}
Copy link
Contributor Author

@vijayraghav-io vijayraghav-io May 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thought of handling non-delegatedTo scenarios.

Comment on lines +220 to +221
}
this.azureUserId = parsedBody.value[0].userPrincipalName ?? undefined;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using UPN instead of GUID

The Microsoft Graph documentation (Outlook change notifications overview) specifies that the {id} in /users/{id}/events can be either:

  • A user principal name (UPN, e.g., user@domain.com).
  • A user’s Azure AD ID (GUID).

Reason for using UPN :

  • It’s more readable and aligns with Cal’s use of email-based identifiers.
  • It’s supported for all Graph API operations, including subscriptions.
  • It avoids the need for additional queries to map GUIDs to users in delegated scenarios.

@@ -52,6 +57,9 @@ interface BodyValue {
start: { dateTime: string };
}

const MICROSOFT_WEBHOOK_URL = `${process.env.NEXT_PUBLIC_WEBAPP_URL}/api/integrations/office365calendar/webhook`;
const MICROSOFT_SUBSCRIPTION_TTL = 3 * 24 * 60 * 60 * 1000; // 3 days in milliseconds
Copy link
Contributor Author

@vijayraghav-io vijayraghav-io May 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

recommended TTL for calendar event subscriptions is 4320 mins (~ 3 Days)
Ref - https://learn.microsoft.com/en-us/answers/questions/2184864/wrong-max-subscription-expiration-on-meetingcallev

hbjORbj
hbjORbj previously requested changes May 2, 2025
Copy link
Contributor

@hbjORbj hbjORbj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was quick! Can you add some unit tests please?

@vijayraghav-io
Copy link
Contributor Author

vijayraghav-io commented May 3, 2025

That was quick! Can you add some unit tests please?

Sure will add 🙏

@coderabbitai coderabbitai bot mentioned this pull request Aug 6, 2025
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🔭 Outside diff range comments (1)
packages/app-store/office365calendar/lib/CalendarService.ts (1)

348-390: Add input validation for date range and calendar IDs.

The method should validate inputs before processing to prevent unnecessary API calls and potential errors.

Add validation at the beginning of the method:

 async fetchAvailability(
   dateFrom: string,
   dateTo: string,
   calendarIds: string[],
   getCreatedDateTime = false
 ): Promise<EventBusyDate[]> {
+  if (!calendarIds || calendarIds.length === 0) {
+    return [];
+  }
+  
   const dateFromParsed = new Date(dateFrom);
   const dateToParsed = new Date(dateTo);
+  
+  if (isNaN(dateFromParsed.getTime()) || isNaN(dateToParsed.getTime())) {
+    throw new Error("Invalid date format provided");
+  }
+  
+  if (dateFromParsed >= dateToParsed) {
+    throw new Error("dateFrom must be before dateTo");
+  }
♻️ Duplicate comments (2)
packages/app-store/office365calendar/lib/CalendarService.ts (2)

697-699: LGTM! Well-documented subscription scoping.

The implementation correctly scopes subscriptions to specific calendars using /calendars/{calendar_id}/events to reduce notification volume, as documented in the Microsoft Graph API guidelines.


700-700: LGTM! Using recommended TTL for calendar subscriptions.

The 3-day TTL aligns with Microsoft's recommended duration for calendar event subscriptions as per the documentation.

🧹 Nitpick comments (3)
packages/app-store/office365calendar/lib/CalendarService.ts (3)

405-436: Consider implementing cache warming strategy.

The cache miss path fetches data but doesn't automatically populate the cache. Consider warming the cache after a miss to benefit subsequent requests.

Consider updating the cache after a miss:

 this.log.debug("[Cache Miss] Fetching availability", { dateFrom, dateTo, calendarIds });
 const data = await this.fetchAvailability(dateFrom, dateTo, calendarIds);
+
+// Optionally warm the cache for future requests
+await this.setAvailabilityInCache(cacheArgs, data).catch((err) => {
+  this.log.warn("Failed to warm cache after miss", err);
+});
+
 return data;

712-726: Add error handling for failed subscription deletions.

While the method logs warnings for failed deletions, consider tracking which deletions failed and potentially retrying or escalating critical failures.

Consider tracking failed deletions:

 private async stopWatchingCalendarsInMicrosoft(subscriptions: { subscriptionId: string | null }[]) {
   const uniqueSubscriptions = subscriptions.filter(
     (s, i, arr) => s.subscriptionId && arr.findIndex((x) => x.subscriptionId === s.subscriptionId) === i
   );

-  await Promise.allSettled(
+  const results = await Promise.allSettled(
     uniqueSubscriptions.map(({ subscriptionId }) =>
       subscriptionId
         ? this.fetcher(`/subscriptions/${subscriptionId}`, { method: "DELETE" }).catch((err) => {
             this.log.warn(`Failed to delete subscription ${subscriptionId}`, err);
+            throw err; // Re-throw to capture in allSettled
           })
         : Promise.resolve()
     )
   );
+  
+  const failedDeletions = results.filter(r => r.status === 'rejected');
+  if (failedDeletions.length > 0) {
+    this.log.error(`Failed to delete ${failedDeletions.length} subscriptions`);
+  }
 }

728-756: Consider batching cache operations for performance.

The method processes calendars per event type sequentially. For better performance with many event types, consider batching the cache operations.

Consider parallel processing with batching:

-for (const [_eventTypeId, selectedCalendars] of Array.from(selectedCalendarsPerEventType.entries())) {
-  const parsedArgs = {
-    timeMin: getTimeMin(),
-    timeMax: getTimeMax(),
-    items: selectedCalendars.map((sc) => ({ id: sc.externalId })),
-  };
-  const data = await this.fetchAvailability(
-    parsedArgs.timeMin,
-    parsedArgs.timeMax,
-    parsedArgs.items.map((i) => i.id)
-  );
-  await this.setAvailabilityInCache(parsedArgs, data);
-}
+const cacheOperations = Array.from(selectedCalendarsPerEventType.entries()).map(
+  async ([_eventTypeId, selectedCalendars]) => {
+    const parsedArgs = {
+      timeMin: getTimeMin(),
+      timeMax: getTimeMax(),
+      items: selectedCalendars.map((sc) => ({ id: sc.externalId })),
+    };
+    const data = await this.fetchAvailability(
+      parsedArgs.timeMin,
+      parsedArgs.timeMax,
+      parsedArgs.items.map((i) => i.id)
+    );
+    return this.setAvailabilityInCache(parsedArgs, data);
+  }
+);
+
+await Promise.all(cacheOperations);
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 8c201de and e2aadb2.

📒 Files selected for processing (1)
  • packages/app-store/office365calendar/lib/CalendarService.ts (9 hunks)
🧰 Additional context used
📓 Path-based instructions (3)
**/*Service.ts

📄 CodeRabbit Inference Engine (.cursor/rules/review.mdc)

Service files must include Service suffix, use PascalCase matching exported class, and avoid generic names (e.g., MembershipService.ts)

Files:

  • packages/app-store/office365calendar/lib/CalendarService.ts
**/*.ts

📄 CodeRabbit Inference Engine (.cursor/rules/review.mdc)

**/*.ts: For Prisma queries, only select data you need; never use include, always use select
Ensure the credential.key field is never returned from tRPC endpoints or APIs

Files:

  • packages/app-store/office365calendar/lib/CalendarService.ts
**/*.{ts,tsx}

📄 CodeRabbit Inference Engine (.cursor/rules/review.mdc)

Flag excessive Day.js use in performance-critical code; prefer native Date or Day.js .utc() in hot paths like loops

Files:

  • packages/app-store/office365calendar/lib/CalendarService.ts
🧠 Learnings (5)
📓 Common learnings
Learnt from: vijayraghav-io
PR: calcom/cal.com#21072
File: packages/app-store/office365calendar/api/webhook.ts:120-123
Timestamp: 2025-07-18T17:57:16.395Z
Learning: The office365calendar webhook handler in packages/app-store/office365calendar/api/webhook.ts is specifically designed for Office365 calendar integration, not as a generic webhook handler. Therefore, it's safe to assume that fetchAvailabilityAndSetCache method will be implemented in the Office365CalendarService, making explicit validation checks unnecessary.
📚 Learning: 2025-08-05T12:04:29.037Z
Learnt from: din-prajapati
PR: calcom/cal.com#21854
File: packages/app-store/office365calendar/__tests__/unit_tests/SubscriptionManager.test.ts:0-0
Timestamp: 2025-08-05T12:04:29.037Z
Learning: In packages/app-store/office365calendar/lib/CalendarService.ts, the fetcher method in Office365CalendarService class is public, not private. It was specifically changed from private to public in this PR to support proper testing and external access patterns.

Applied to files:

  • packages/app-store/office365calendar/lib/CalendarService.ts
📚 Learning: 2025-07-18T17:57:16.395Z
Learnt from: vijayraghav-io
PR: calcom/cal.com#21072
File: packages/app-store/office365calendar/api/webhook.ts:120-123
Timestamp: 2025-07-18T17:57:16.395Z
Learning: The office365calendar webhook handler in packages/app-store/office365calendar/api/webhook.ts is specifically designed for Office365 calendar integration, not as a generic webhook handler. Therefore, it's safe to assume that fetchAvailabilityAndSetCache method will be implemented in the Office365CalendarService, making explicit validation checks unnecessary.

Applied to files:

  • packages/app-store/office365calendar/lib/CalendarService.ts
📚 Learning: 2025-07-18T08:47:01.264Z
Learnt from: vijayraghav-io
PR: calcom/cal.com#21072
File: packages/prisma/schema.prisma:891-891
Timestamp: 2025-07-18T08:47:01.264Z
Learning: The Outlook Calendar integration in Cal.com intentionally reuses subscription IDs across multiple event types for efficiency. The `upsertSelectedCalendarsForEventTypeIds` method creates separate SelectedCalendar records for each eventTypeId, all sharing the same outlookSubscriptionId. This subscription sharing pattern means that unique constraints like `@unique([outlookSubscriptionId, eventTypeId])` should not be applied as they would prevent this intended functionality.

Applied to files:

  • packages/app-store/office365calendar/lib/CalendarService.ts
📚 Learning: 2025-07-18T08:47:01.264Z
Learnt from: vijayraghav-io
PR: calcom/cal.com#21072
File: packages/prisma/schema.prisma:891-891
Timestamp: 2025-07-18T08:47:01.264Z
Learning: In Cal.com's calendar integration, both Google Calendar and Outlook Calendar are designed to allow multiple eventTypeIds to share the same subscription ID (googleChannelId or outlookSubscriptionId). This is an intentional design pattern to reuse existing subscriptions for efficiency rather than creating separate subscriptions for each event type. Therefore, unique constraints like `@unique([outlookSubscriptionId, eventTypeId])` should not be added as they would break this subscription sharing functionality.

Applied to files:

  • packages/app-store/office365calendar/lib/CalendarService.ts
🧬 Code Graph Analysis (1)
packages/app-store/office365calendar/lib/CalendarService.ts (4)
packages/types/Calendar.d.ts (3)
  • EventBusyDate (56-62)
  • IntegrationCalendar (247-256)
  • SelectedCalendarEventTypeIds (261-261)
packages/features/calendar-cache/calendar-cache.ts (1)
  • CalendarCache (15-31)
packages/features/calendar-cache/lib/datesForCache.ts (2)
  • getTimeMin (2-8)
  • getTimeMax (21-59)
packages/lib/server/repository/selectedCalendar.ts (1)
  • SelectedCalendarRepository (48-502)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: Install dependencies / Yarn install & cache
🔇 Additional comments (3)
packages/app-store/office365calendar/lib/CalendarService.ts (3)

177-185: LGTM! Proper handling of non-delegated scenarios.

The implementation correctly fetches the UserPrincipalName (UPN) for non-delegated users using the /me endpoint and includes appropriate error handling when the UPN is missing.


208-230: LGTM! Well-documented implementation using UPN.

The implementation correctly uses UserPrincipalName (UPN) instead of GUID, which aligns with Microsoft Graph documentation and improves readability. The regex pattern for removing OAuth client ID aliases is appropriate.


1003-1021: LGTM! Clean implementation of busy time processing.

The method correctly filters out free/workingElsewhere events and properly enriches the response with optional id and createdDateTime fields when available.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🙋 Bounty claim 💎 Bounty A bounty on Algora.io calendar-apps area: calendar, google calendar, outlook, lark, microsoft 365, apple calendar community Created by Linear-GitHub Sync ❗️ .env changes contains changes to env variables ✨ feature New feature or request High priority Created by Linear-GitHub Sync high-risk Requires approval by Foundation team ❗️ migrations contains migration files teams area: teams, round robin, collective, managed event-types webhooks area: webhooks, callback, webhook payload $500
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Outlook Cache – Bounty-to-Hire
5 participants