Skip to content

feat: Read params from file for template/workspace creation #1541

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
May 20, 2022
Merged

feat: Read params from file for template/workspace creation #1541

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
b35b7b3
Read params from file for template/workspace creation
AbhineetJain May 18, 2022
e69ae31
Use os.ReadFile
AbhineetJain May 18, 2022
dc80543
Refactor reading params into a separate module
AbhineetJain May 19, 2022
931e7d5
Add comments and unit tests
AbhineetJain May 20, 2022
0e6b2d4
Fix merge conflicts and refactor
AbhineetJain May 20, 2022
8a075f4
Rename variable
AbhineetJain May 20, 2022
71e94e3
Uncomment and fix unit test
AbhineetJain May 20, 2022
d27d202
Fix comment
AbhineetJain May 20, 2022
692b975
Refactor tests
AbhineetJain May 20, 2022
1a12be1
Fix unit tests for windows
AbhineetJain May 20, 2022
1491620
Fix unit tests for Windows
AbhineetJain May 20, 2022
74989ac
Add comments for the hotfix
AbhineetJain May 20, 2022
e584e81
Merge branch 'main' into 1377-feat-load-parameters-from-file
AbhineetJain May 20, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 17 additions & 5 deletions cli/create.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ func create() *cobra.Command {
var (
workspaceName string
templateName string
parameterFile string
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tfvars is the standard way to handle this with Terraform.

I'd prefer to not have our own solution, because that'd remove the ability for users to do terraform apply.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kylecarbs that assumes we use terraform, but our cli should be terraform agnostic right?

Also tfvars is just HCL or yaml. So we could just support multiple file types

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Developers also won't have the tf file when running coder workspace create, only the person who made the template will have the tf file.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can be Terraform agnostic while still consuming tfvars. Latching onto the existing Terraform ecosystem is a much better user experience than forking it with our own parameter format.

I don't believe it's necessary to have a tfvars equivalent when creating a workspace. Or at least, we haven't seen it as a problem yet.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issue #1377 mentions creating workspaces using parameter files, which is why I went with a non-tfvars approach. However, #1426 does talk about doing this for templates. Would you recommend solving both problems using different approaches or only work on #1426 (template creation via file) via tfvars? Let me quickly check how feasible using tfvars for template creation is.

Copy link
Contributor Author

@AbhineetJain AbhineetJain May 18, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a documented flow of how the customers would use terraform apply when setting up templates? I tried using the tfvars file as follows:

$ terraform init
$ terraform apply -var-file="<path_to_tfvars>"
...
Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

It did not create a template and calling coder templates create after this still asked me for input, so I am not sure what the actual flow should be.

)
cmd := &cobra.Command{
Annotations: workspaceCommand,
Expand Down Expand Up @@ -116,23 +117,33 @@ func create() *cobra.Command {
return err
}

printed := false
// parameterMapFromFile can be nil if parameter file is not specified
var parameterMapFromFile map[string]string
if parameterFile != "" {
_, _ = fmt.Fprintln(cmd.OutOrStdout(), cliui.Styles.Paragraph.Render("Attempting to read the variables from the parameter file.")+"\r\n")
parameterMapFromFile, err = createParameterMapFromFile(parameterFile)
if err != nil {
return err
}
}

disclaimerPrinted := false
parameters := make([]codersdk.CreateParameterRequest, 0)
for _, parameterSchema := range parameterSchemas {
if !parameterSchema.AllowOverrideSource {
continue
}
if !printed {
if !disclaimerPrinted {
_, _ = fmt.Fprintln(cmd.OutOrStdout(), cliui.Styles.Paragraph.Render("This template has customizable parameters. Values can be changed after create, but may have unintended side effects (like data loss).")+"\r\n")
printed = true
disclaimerPrinted = true
}
value, err := cliui.ParameterSchema(cmd, parameterSchema)
parameterValue, err := getParameterValueFromMapOrInput(cmd, parameterMapFromFile, parameterSchema)
if err != nil {
return err
}
parameters = append(parameters, codersdk.CreateParameterRequest{
Name: parameterSchema.Name,
SourceValue: value,
SourceValue: parameterValue,
SourceScheme: codersdk.ParameterSourceSchemeData,
DestinationScheme: parameterSchema.DefaultDestinationScheme,
})
Expand Down Expand Up @@ -194,5 +205,6 @@ func create() *cobra.Command {
}

cliflag.StringVarP(cmd.Flags(), &templateName, "template", "t", "CODER_TEMPLATE_NAME", "", "Specify a template name.")
cliflag.StringVarP(cmd.Flags(), &parameterFile, "parameter-file", "", "CODER_PARAMETER_FILE", "", "Specify a file path with parameter values.")
return cmd
}
144 changes: 111 additions & 33 deletions cli/create_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ package cli_test

import (
"fmt"
"os"
"testing"

"github.com/stretchr/testify/require"
Expand Down Expand Up @@ -113,39 +114,7 @@ func TestCreate(t *testing.T) {

defaultValue := "something"
version := coderdtest.CreateTemplateVersion(t, client, user.OrganizationID, &echo.Responses{
Parse: []*proto.Parse_Response{{
Type: &proto.Parse_Response_Complete{
Complete: &proto.Parse_Complete{
ParameterSchemas: []*proto.ParameterSchema{
{
AllowOverrideSource: true,
Name: "region",
Description: "description 1",
DefaultSource: &proto.ParameterSource{
Scheme: proto.ParameterSource_DATA,
Value: defaultValue,
},
DefaultDestination: &proto.ParameterDestination{
Scheme: proto.ParameterDestination_PROVISIONER_VARIABLE,
},
},
{
AllowOverrideSource: true,
Name: "username",
Description: "description 2",
DefaultSource: &proto.ParameterSource{
Scheme: proto.ParameterSource_DATA,
// No default value
Value: "",
},
DefaultDestination: &proto.ParameterDestination{
Scheme: proto.ParameterDestination_PROVISIONER_VARIABLE,
},
},
},
},
},
}},
Parse: createTestParseResponseWithDefault(defaultValue),
Provision: echo.ProvisionComplete,
ProvisionDryRun: echo.ProvisionComplete,
})
Expand Down Expand Up @@ -178,4 +147,113 @@ func TestCreate(t *testing.T) {
}
<-doneChan
})

t.Run("WithParameterFileContainingTheValue", func(t *testing.T) {
t.Parallel()
client := coderdtest.New(t, &coderdtest.Options{IncludeProvisionerD: true})
user := coderdtest.CreateFirstUser(t, client)

defaultValue := "something"
version := coderdtest.CreateTemplateVersion(t, client, user.OrganizationID, &echo.Responses{
Parse: createTestParseResponseWithDefault(defaultValue),
Provision: echo.ProvisionComplete,
ProvisionDryRun: echo.ProvisionComplete,
})

coderdtest.AwaitTemplateVersionJob(t, client, version.ID)
_ = coderdtest.CreateTemplate(t, client, user.OrganizationID, version.ID)
tempDir := t.TempDir()
parameterFile, _ := os.CreateTemp(tempDir, "testParameterFile*.yaml")
_, _ = parameterFile.WriteString("region: \"bingo\"\nusername: \"boingo\"")
cmd, root := clitest.New(t, "create", "", "--parameter-file", parameterFile.Name())
clitest.SetupConfig(t, client, root)
doneChan := make(chan struct{})
pty := ptytest.New(t)
cmd.SetIn(pty.Input())
cmd.SetOut(pty.Output())
go func() {
defer close(doneChan)
err := cmd.Execute()
require.NoError(t, err)
}()

matches := []string{
"Specify a name", "my-workspace",
"Confirm create?", "yes",
}
for i := 0; i < len(matches); i += 2 {
match := matches[i]
value := matches[i+1]
pty.ExpectMatch(match)
pty.WriteLine(value)
}
<-doneChan
removeTmpDirUntilSuccess(t, tempDir)
})
t.Run("WithParameterFileNotContainingTheValue", func(t *testing.T) {
t.Parallel()
client := coderdtest.New(t, &coderdtest.Options{IncludeProvisionerD: true})
user := coderdtest.CreateFirstUser(t, client)

defaultValue := "something"
version := coderdtest.CreateTemplateVersion(t, client, user.OrganizationID, &echo.Responses{
Parse: createTestParseResponseWithDefault(defaultValue),
Provision: echo.ProvisionComplete,
ProvisionDryRun: echo.ProvisionComplete,
})
coderdtest.AwaitTemplateVersionJob(t, client, version.ID)
template := coderdtest.CreateTemplate(t, client, user.OrganizationID, version.ID)
tempDir := t.TempDir()
parameterFile, _ := os.CreateTemp(tempDir, "testParameterFile*.yaml")
_, _ = parameterFile.WriteString("zone: \"bananas\"")
cmd, root := clitest.New(t, "create", "my-workspace", "--template", template.Name, "--parameter-file", parameterFile.Name())
clitest.SetupConfig(t, client, root)
doneChan := make(chan struct{})
pty := ptytest.New(t)
cmd.SetIn(pty.Input())
cmd.SetOut(pty.Output())
go func() {
defer close(doneChan)
err := cmd.Execute()
require.EqualError(t, err, "Parameter value absent in parameter file for \"region\"!")
}()
<-doneChan
removeTmpDirUntilSuccess(t, tempDir)
})
}

func createTestParseResponseWithDefault(defaultValue string) []*proto.Parse_Response {
return []*proto.Parse_Response{{
Type: &proto.Parse_Response_Complete{
Complete: &proto.Parse_Complete{
ParameterSchemas: []*proto.ParameterSchema{
{
AllowOverrideSource: true,
Name: "region",
Description: "description 1",
DefaultSource: &proto.ParameterSource{
Scheme: proto.ParameterSource_DATA,
Value: defaultValue,
},
DefaultDestination: &proto.ParameterDestination{
Scheme: proto.ParameterDestination_PROVISIONER_VARIABLE,
},
},
{
AllowOverrideSource: true,
Name: "username",
Description: "description 2",
DefaultSource: &proto.ParameterSource{
Scheme: proto.ParameterSource_DATA,
// No default value
Value: "",
},
DefaultDestination: &proto.ParameterDestination{
Scheme: proto.ParameterDestination_PROVISIONER_VARIABLE,
},
},
},
},
},
}}
}
56 changes: 56 additions & 0 deletions cli/parameter.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
package cli

import (
"os"

"golang.org/x/xerrors"
"gopkg.in/yaml.v3"

"github.com/coder/coder/cli/cliui"
"github.com/coder/coder/codersdk"
"github.com/spf13/cobra"
)

// Reads a YAML file and populates a string -> string map.
// Throws an error if the file name is empty.
func createParameterMapFromFile(parameterFile string) (map[string]string, error) {
if parameterFile != "" {
parameterMap := make(map[string]string)

parameterFileContents, err := os.ReadFile(parameterFile)

if err != nil {
return nil, err
}

err = yaml.Unmarshal(parameterFileContents, &parameterMap)

if err != nil {
return nil, err
}

return parameterMap, nil
}

return nil, xerrors.Errorf("Parameter file name is not specified")
}

// Returns a parameter value from a given map, if the map exists, else takes input from the user.
// Throws an error if the map exists but does not include a value for the parameter.
func getParameterValueFromMapOrInput(cmd *cobra.Command, parameterMap map[string]string, parameterSchema codersdk.ParameterSchema) (string, error) {
var parameterValue string
if parameterMap != nil {
var ok bool
parameterValue, ok = parameterMap[parameterSchema.Name]
if !ok {
return "", xerrors.Errorf("Parameter value absent in parameter file for %q!", parameterSchema.Name)
}
} else {
var err error
parameterValue, err = cliui.ParameterSchema(cmd, parameterSchema)
if err != nil {
return "", err
}
}
return parameterValue, nil
}
79 changes: 79 additions & 0 deletions cli/parameter_internal_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
package cli

import (
"os"
"runtime"
"testing"

"github.com/stretchr/testify/assert"
)

func TestCreateParameterMapFromFile(t *testing.T) {
t.Parallel()
t.Run("CreateParameterMapFromFile", func(t *testing.T) {
t.Parallel()
tempDir := t.TempDir()
parameterFile, _ := os.CreateTemp(tempDir, "testParameterFile*.yaml")
_, _ = parameterFile.WriteString("region: \"bananas\"\ndisk: \"20\"\n")

parameterMapFromFile, err := createParameterMapFromFile(parameterFile.Name())

expectedMap := map[string]string{
"region": "bananas",
"disk": "20",
}

assert.Equal(t, expectedMap, parameterMapFromFile)
assert.Nil(t, err)

removeTmpDirUntilSuccess(t, tempDir)
})
t.Run("WithEmptyFilename", func(t *testing.T) {
t.Parallel()

parameterMapFromFile, err := createParameterMapFromFile("")

assert.Nil(t, parameterMapFromFile)
assert.EqualError(t, err, "Parameter file name is not specified")
})
t.Run("WithInvalidFilename", func(t *testing.T) {
t.Parallel()

parameterMapFromFile, err := createParameterMapFromFile("invalidFile.yaml")

assert.Nil(t, parameterMapFromFile)

// On Unix based systems, it is: `open invalidFile.yaml: no such file or directory`
// On Windows, it is `open invalidFile.yaml: The system cannot find the file specified.`
if runtime.GOOS == "windows" {
assert.EqualError(t, err, "open invalidFile.yaml: The system cannot find the file specified.")
} else {
assert.EqualError(t, err, "open invalidFile.yaml: no such file or directory")
}
})
t.Run("WithInvalidYAML", func(t *testing.T) {
t.Parallel()
tempDir := t.TempDir()
parameterFile, _ := os.CreateTemp(tempDir, "testParameterFile*.yaml")
_, _ = parameterFile.WriteString("region = \"bananas\"\ndisk = \"20\"\n")

parameterMapFromFile, err := createParameterMapFromFile(parameterFile.Name())

assert.Nil(t, parameterMapFromFile)
assert.EqualError(t, err, "yaml: unmarshal errors:\n line 1: cannot unmarshal !!str `region ...` into map[string]string")

removeTmpDirUntilSuccess(t, tempDir)
})
}

// Need this for Windows because of a known issue with Go:
// https://github.com/golang/go/issues/52986
func removeTmpDirUntilSuccess(t *testing.T, tempDir string) {
t.Helper()
t.Cleanup(func() {
err := os.RemoveAll(tempDir)
for err != nil {
err = os.RemoveAll(tempDir)
}
})
}
Loading