SAML-BOX (https://saml-box.com) is an online platform developed to simplify SAML testing and integration, offering both an Identity Provider (IDP) and a Service Provider (SP).
Developing SAML-BOX is an experiment to see how AI can help code development. This article explores Pros and Cons of coding with AI too.
A lot of times, I want to have an IDP (Identity Provider) or SP (Service Provider) for my SAML testing or development, but couldn't find ones that are easy to use online. On top of that, with the recent boom of AI, I'm curious to see how much more efficient it will help on code development. So I developed this SAML-BOX (https://saml-box.com), a readily available online IDP and SP for SAML testing and integration.
With SAML-BOX, developers can streamline SAML development and integration while leveraging tools like Keycloak and Spring Security SAML. This post explores SAML-BOX’s features and shares some insights from my experience developing with AI.
What is SAML-BOX?
SAML-BOX is a custom-built public platform designed to simplify the development and integration of SAML-based applications. It consists of two main components:
Identity Provider (IDP): Built with Keycloak, an open-source identity management solution, the IDP handles user authentication and provides SAML assertions to service providers.
Service Provider (SP): Powered by Spring Security SAML, the SP processes SAML responses from the IDP, enabling efficient access to secured resources.
Here’s a high-level diagram of the component structure:
IDP with Keycloak
Keycloak is an Open Source Identity and Access Management platform. It adds authentication to applications and secure services with minimum effort, and it provides user federation, strong authentication, user management, fine-grained authorization, etc.
I use Keycloak here as IDP at the backend in SAML BOX. I wouldn't say it's much of a powerful IDP platform, but should serve the purpose here. A lot of the functions are out-of-box by Keycloak and what I need to do mainly is to wrap a portal app in front to provide UI and then call the Keycloak service APIs to manage the settings.
SP with Spring Security SAML
On the SP side, SAML-BOX leverages Spring Security SAML, which provides comprehensive support for working as an SP (though it doesn't support IDP). The detail information can be found here.
Using Spring Security SAML to provide multi-tenant IDP capability is a little tricky to do. The framework out-of-the-box is for SP (Spring) to integrate with a target IDP. To support multiple IDPs, it will need to be tweaked. The good thing is Spring Security SAML does have some customization capability, while the challenge here is how to actually implement the customization without proper documentation.
As I was experimenting with AI tools during SAML-BOX development, I want to share some thoughts on AI too.
The Role of AI in Enhancing Development Efficiency
AI has gained immense popularity in recent years, and I’ve used SAML-BOX as an opportunity to explore how it can enhance code development.
AI Tools Used
During SAML-BOX’s development, I experimented with AI tools like DeepSeek, ChatGPT, Grok, Gemini, and Claude to gauge their coding capabilities. Overall, they perform similarly, though some are slightly “smarter” or more user-friendly than others. I primarily relied on DeepSeek because I appreciated how it clearly outlines its thought process.
Pros of Development with AI
Here are the key benefits I discovered while coding with AI.
Code Bootstrapping
This is one great advantage of coding with AI. Whenever you have an idea for something you want to develop, you can ask AI to give you a solution with proper prompting. Although 90% of the time the solution won't work directly, it does provide something for you to work with. And depending on the complexity of the ask, the amount of change you need to make will vary. Sometimes it would be a small tweak to get the whole code to work, while sometimes you need to dig quite a bit to get it roll.
For example, when I was working with the SP part using Spring Security SAML, I simply started by asking DeepSeek something like "I want to use Spring Security SAML to build an SP application where it can integrates with multiple IDPs." It will then generate all the code that I need to start to work on.
Troubleshooting
AI can also help on troubleshooting. Whenever I'm getting an error, either at compile time or run time, I will copy and paste the error message into AI and it will give you a lot of useful information for troubleshooting. Sometimes, it can directly pinpoint the issue and provide a fix; other times, it will give you analysis and a list of strategies on how to troubleshoot. In general, I find those helpful on fixing issues.
Enhancing Existing Code
AI can also improve existing code. For instance, I had an HTML template with JavaScript and CSS and wanted to add Bootstrap styling. I pasted the code into DeepSeek, and it returned an updated version with decent styling applied.
These are the primary benefits I experienced, though there are others as well.
Cons of Development with AI
Despite its advantages, AI has notable disadvantages too.
Context Limitation
Perhaps one of the biggest limitations of AI coding is that it's difficult for AI to understand the code as a whole, or in other words, putting in context when generating/updating code.
Imagine a case like this. There is a project your team has been working on and the code repo has grown to a decent size. Now, you want to add a feature or function on top of the current repo, and this is assuming the new piece of code added should have the context knowledge of existing code repo. That's by default when we are coding as human, but not the case for AI.
There are tools out there that can now scan the repo and provide suggestions to enhance it, but that's still different from working with general AI for development.
Can’t Provide a Solution
AI will never tell you directly that it can't provide you a solution. Instead, it will always throw you something as if it works. It's good if you can figure out a fix for it, but it will be frustrating if you have tried a lot and still can't fix it. This is specially true when you are dealing with something that's high customized and requires some 'deep' domain knowledge. After all, AI is trained with available data out there. The more customized and 'deep', the less data are available.
For example, I was stuck at an issue when customization is needed for parsing SAML response in Spring Security SAML. Tried different tools, but whatever AI provided wouldn't work and neither it can provide a base for work, as the base itself doesn't look correct. Eventually, I have to go through the Spring Security documentation with plenty of details and try to figure out a way to do it.
Again, there are other limitations as well, but the two items mentioned above are currently crucial.
Final Thoughts
There are a lot of voices out there praising on AI and coding. It looks fancy and promising in the future, but for now, I feel like this is still at its early stage and will take time or even require some underlying breakthrough before it can be really useful. Considering the its limitations, especially the ones I mentioned above, the disadvantages has outweighs the advantages it has in the industry-level practices (One a side note is that IDE tools based on AI e.g. cursor, is helpful for coding, but that's at a different perspective).
Yet, having said the above, why do people still believe in AI given its limitations. One of the reasons might be its potential. Whatever can't be done right now doesn't necessarily mean it can't be done in the future either. Big Countries around the world has marked AI development as national strategies and huge capitals are poured in. At least I think this AI race will be exciting for the next decades.